Smartglasses for detecting congestive heart failure

ABSTRACT

Systems for calculating extent of congestive heart failure (CHF) and/or identifying exacerbation of CHF. In one embodiment, a system includes smartglasses configured to be worn on a user&#39;s head, and an inward-facing camera and a sensor, both physically coupled to the smartglasses. The inward-facing camera is mounted more than 5 mm away from the head and captures images of an area comprising skin on the user&#39;s head, which is larger than 4 cm{circumflex over ( )}2. The sensor measures a signal indicative of a respiration rate of the user (respiration signal). The system also includes a computer that calculates the extent of CHF based on: a facial blood flow pattern recognizable in the images, and respiration rate recognizable in the respiration signal.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application claims priority to U.S. Provisional Patent ApplicationNo. 62/874,430, filed Jul. 15, 2019, U.S. Provisional Patent ApplicationNo. 62/928,726, filed Oct. 31, 2019, U.S. Provisional Patent ApplicationNo. 62/945,141, filed Dec. 7, 2019, and U.S. Provisional PatentApplication No. 62/960,913, filed Jan. 14, 2020.

This Application is a Continuation-In-Part of U.S. application Ser. No.16/551,654, filed Aug. 26, 2019. U.S. Ser. No. 16/551,654 is aContinuation-In-Part of U.S. application Ser. No. 16/453,993, filed Jun.26, 2019. U.S. Ser. No. 16/453,993 is a Continuation-In-Part of U.S.application Ser. No. 16/375,841, filed Apr. 4, 2019. U.S. Ser. No.16/375,841 is a Continuation-In-Part of U.S. application Ser. No.16/156,493, filed Oct. 10, 2018. U.S. Ser. No. 16/156,493, is aContinuation-In-Part of U.S. application Ser. No. 15/635,178, filed Jun.27, 2017, now U.S. Pat. No. 10,136,856, which claims priority to U.S.Provisional Patent Application No. 62/354,833, filed Jun. 27, 2016, andU.S. Provisional Patent Application No. 62/372,063, filed Aug. 8, 2016.

U.S. Ser. No. 16/156,493 is also a Continuation-In-Part of U.S.application Ser. No. 15/231,276, filed Aug. 8, 2016, which claimspriority to U.S. Provisional Patent Application No. 62/202,808, filedAug. 8, 2015, and U.S. Provisional Patent Application No. 62/236,868,filed Oct. 3, 2015.

U.S. Ser. No. 16/156,493 is also a Continuation-In-Part of U.S.application Ser. No. 15/832,855, filed Dec. 6, 2017, now U.S. Pat. No.10,130,308, which claims priority to U.S. Provisional Patent ApplicationNo. 62/456,105, filed Feb. 7, 2017, and U.S. Provisional PatentApplication No. 62/480,496, filed Apr. 2, 2017, and U.S. ProvisionalPatent Application No. 62/566,572, filed Oct. 2, 2017. U.S. Ser. No.15/832,855 is a Continuation-In-Part of U.S. application Ser. No.15/182,592, filed Jun. 14, 2016, now U.S. Pat. No. 10,165,949, aContinuation-In-Part of U.S. application Ser. No. 15/231,276, filed Aug.8, 2016, a Continuation-In-Part of U.S. application Ser. No. 15/284,528,filed Oct. 3, 2016, now U.S. Pat. No. 10,113,913, a Continuation-In-Partof U.S. application Ser. No. 15/635,178, filed Jun. 27, 2017, now U.S.Pat. No. 10,136,856, and a Continuation-In-Part of U.S. application Ser.No. 15/722,434, filed Oct. 2, 2017.

U.S. Ser. No. 15/832,855 is a Continuation-In-Part of U.S. applicationSer. No. 15/182,566, filed Jun. 14, 2016, now U.S. Pat. No. 9,867,546,which claims priority to U.S. Provisional Patent Application No.62/175,319, filed Jun. 14, 2015, and U.S. Provisional Patent ApplicationNo. 62/202,808, filed Aug. 8, 2015.

U.S. Ser. No. 15/182,592 claims priority to U.S. Provisional PatentApplication No. 62/175,319, filed Jun. 14, 2015, and U.S. ProvisionalPatent Application No. 62/202,808, filed Aug. 8, 2015.

U.S. Ser. No. 15/284,528 claims priority to U.S. Provisional PatentApplication No. 62/236,868, filed Oct. 3, 2015, and U.S. ProvisionalPatent Application No. 62/354,833, filed Jun. 27, 2016, and U.S.Provisional Patent Application No. 62/372,063, filed Aug. 8, 2016.

U.S. Ser. No. 16/156,493 is also a Continuation-In-Part of U.S.application Ser. No. 15/833,115, filed Dec. 6, 2017, now U.S. Pat. No.10,130,261. U.S. Ser. No. 15/833,115 is a Continuation-In-Part of U.S.application Ser. No. 15/182,592, a Continuation-In-Part of U.S.application Ser. No. 15/231,276, filed Aug. 8, 2016, aContinuation-In-Part of U.S. application Ser. No. 15/284,528, aContinuation-In-Part of U.S. application Ser. No. 15/635,178, and aContinuation-In-Part of U.S. application Ser. No. 15/722,434, filed Oct.2, 2017.

U.S. Ser. No. 16/453,993 is also a Continuation-In-Part of U.S.application Ser. No. 16/147,695, filed Sep. 29, 2018. U.S. Ser. No.16/147,695 is a Continuation of U.S. application Ser. No. 15/182,592,filed Jun. 14, 2016, which claims priority to U.S. Provisional PatentApplication No. 62/175,319, filed Jun. 14, 2015, and U.S. ProvisionalPatent Application No. 62/202,808, filed Aug. 8, 2015.

This Application is a Continuation-In-Part of U.S. Ser. No. 16/156,586,filed Oct. 10, 2018, that is a Continuation-In-Part of U.S. applicationSer. No. 15/832,815, filed Dec. 6, 2017, which claims priority to U.S.Provisional Patent Application No. 62/456,105, filed Feb. 7, 2017, andU.S. Provisional Patent Application No. 62/480,496, filed Apr. 2, 2017,and U.S. Provisional Patent Application No. 62/566,572, filed Oct. 2,2017. U.S. Ser. No. 16/156,586 is also a Continuation-In-Part of U.S.application Ser. No. 15/859,772 Jan. 2, 2018, now U.S. Pat. No.10,159,411.

ACKNOWLEDGMENTS

Gil Thieberger would like to thank his holy and beloved teacher, LamaDvora-hla, for her extraordinary teachings and manifestation of wisdom,love, compassion and morality, and for her endless efforts, support, andskills in guiding him and others on their paths to freedom and ultimatehappiness. Gil would also like to thank his beloved parents for raisinghim with love and care.

BACKGROUND

Heart failure, also known as congestive heart failure (CHF), occurs whenthe heart muscle does not pump blood as well as needed. Certainconditions, such as narrowed arteries in the heart (coronary arterydisease) or high blood pressure, gradually leave the heart too weak ortoo stiff to fill and pump efficiently. CHF is a growing public healthproblem that affects nearly 6.5 million individuals in the US, and 26million individuals worldwide. CHF is known to cause morehospitalizations in people over 65 than pneumonia and heart attacks.

Currently, drug therapy is the mainstay of treatment for CHF. However,without proper monitoring, treatment of CHF relies primarily on crisisintervention. With proper monitoring, more proactive and preventativedisease management approaches may be attempted, which may reduce thenumber of hospitalizations and improve patient outcomes. However,monitoring CHF patients often relies on scheduled visits to a hospitalfollowing up on a cardiac event, home monitoring visits by nurses, andpatient's self-monitoring performed at home. Such monitoring may oftenbe expensive or inconvenient. Self-monitoring at home may require usingspecialized equipment (e.g., monitoring signals from implanteddefibrillators), or require patients to periodically measure their vitalsigns and answer questions related to shortness of breath and fatigue.Thus, there is need for better CHF monitoring approaches that willenable convenient and more continuous monitoring of CHF patients inorder to be able to deliver effective and timely interventions.

SUMMARY

Congestive heart failure (CHF) results in an inability of the heart topump blood efficiently to meet the physiological demands of day-to-dayactivities. This causes manifestation of various signs related to theinability to adequately perform physical activities. These signs mayinclude: shortness of breath and an increase in the respiration rate, anincreased and/or irregular heart rate, a decrease in blood flowthroughout the body which affects the patient's facial blood flowpattern, and a decrease in physical activity. Detection of these signs,the severity of their manifestation, and/or the change to their severitycompared to a baseline, can be used to calculate an extent to which aperson is experiencing CHF.

Some aspects of this disclosure involve utilization of sensors that arephysically coupled to smartglasses in order to conveniently, andoptionally continuously, monitor CHF patients. Smartglasses aregenerally comfortable to wear, lightweight, and can have extendedbattery life. Thus, they are well suited as an instrument for long-termmonitoring of patient's physiological signals and activity, in order todetermine an extent to which the patient suffers from CHF and/or whetherthe patient is experiencing an exacerbation of CHF.

Some embodiments described herein utilize head-mounted sensors to obtainimages of an area on a user's head. These images may be indicative of ablood flow pattern in the area on the user's head. For example, theimages may be indicative of flushness in the area and/or includeinformation from which an imaging photoplethysmogram signal (iPPGsignal) may be obtained. A photoplethysmogram signal (PPG signal) is anoptically obtained plethysmogram that is indicative of blood volumechanges in the microvascular bed of tissue. A PPG signal is oftenobtained by using a pulse oximeter, which illuminates the skin andmeasures changes in light absorption. Another possibility for obtainingthe PPG signal is using an imaging photoplethysmography (iPPG) device.As opposed to contact PPG devices, iPPG does not require contact withthe skin and is obtained by a non-contact sensor, such as a videocamera.

One aspect of this disclosure involves a system configured to calculateextent of congestive heart failure (CHF). In some embodiments, thesystem includes smartglasses configured to be worn on a user's head andan inward-facing camera, physically coupled to the smartglasses. Theinward-facing camera captures images of an area that includes skin onthe user's head. Optionally, the area is larger than 4 cm{circumflexover ( )}2, and the camera is mounted more than 5 mm away from the head.Optionally, the area includes a portion of the user's lips. The systemalso includes a sensor and a computer. The sensor, which is alsophysically coupled to the smartglasses, measures a signal indicative ofa respiration rate of the user (referred to herein as respirationsignal). The computer calculates the extent of CHF based on: a facialblood flow pattern recognizable in the images, and respiration raterecognizable in the respiration signal. Optionally, the images and therespiration signal are measured over a period spanning multiple days,and the computer may also identify an exacerbation of the CHF based on:a reduction in average facial blood flow recognizable in the imagestaken during the period, and an increase in an average respiration raterecognizable in the respiration signal measured during the period.

In one embodiment, the images and respiration signal were measured whilethe user was at rest and prior to a period during which the user walked.Optionally, the computer receives: (i) additional images, taken withinten minutes after the period with the inward-facing camera, and (ii) anadditional signal indicative of a previous respiration rate of the user(additional respiration signal), measured with the sensor within tenminutes after the period. Optionally, the computer is calculates theextent of CHF based on: a difference between a facial blood flow patternrecognizable in the images and an additional facial blood flow patternrecognizable in the additional images, and a difference between arespiration rate recognizable in the respiration signal and the previousrespiration rate recognizable in the additional respiration signal.

In one embodiment, the system includes a movement sensor, physicallycoupled to the smartglasses, and configured to measure movements of theuser. In this embodiment, the computer calculates a number of stepsperformed by the user during the period, and calculates the extent ofCHF responsive to the number of steps exceeding a predeterminedthreshold greater than twenty steps.

In another embodiment, the computer calculates a value indicative ofskin color at different times based on the additional images, and tocalculate the extent of CHF based on a length of a duration followingthe period, in which the difference between the skin color and abaseline skin color, calculated based on the images, was above athreshold.

In yet another embodiment, the computer calculates a value indicative ofskin color at different times based on the additional images, andcalculates the extent of CHF based on a rate of return of the user'sskin color to a baseline skin color calculated based on the images.

In still another embodiment, the computer calculates respiration ratesof the user at different times based on the additional respirationsignals, and calculates the extent of CHF based on a length of aduration following the period, in which the difference between therespiration rate of the user and a baseline respiration rate, calculatedbased on the respiration rates, was above a threshold.

The images may be utilized in various ways to calculate the extent ofCHF. In one example, the computer calculates, based on the images, avalue indicative of an extent to which skin in the area is blue and/orgray, and utilizes a difference between the value and a baseline valuefor the extent to which skin in the area is blue and/or gray tocalculate the extent of CHF. Optionally, the baseline value wasdetermined while the user experienced a certain baseline extent of CHF.In another example, the computer calculates, based on the images, avalue indicative of extent of color changes to skin in the area due tocardiac pulses, and utilizes a difference between the value and abaseline value for the extent of the color changes to calculate theextent of CHF. Optionally, the baseline value for the extent of thecolor changes was determined while the user experienced a certainbaseline extent of CHF.

In some embodiments, temperature measurements may be utilized in thecalculation of the extent of CHF and/or identification of exacerbationof CHF. In one embodiment, the system includes a head-mounted sensorthat measures temperature of a region comprising skin on the user's head(T_(skin)). In this embodiment, the computer utilizes T_(skin) tocompensate for effects of skin temperature on the facial blood flowpattern. In another embodiment, the system includes a head-mountedsensor that measures environmental temperature (T_(env)). In thisembodiment, the computer utilizes T_(env) to compensate for effects ofphysiologic changes related to regulating the user's body temperature onthe facial blood flow pattern.

Another aspect of this disclosure involves a system configured toidentify exacerbation of congestive heart failure (CHF). In someembodiments, the system includes smartglasses configured to be worn on auser's head, an inward-facing camera and sensor, which physicallycoupled to the smartglasses, and a computer. The inward-facing cameracaptures images of an area comprising skin on the user's head, which areindicative of a facial blood flow pattern of the user. Optionally, thearea is larger than 4 cm{circumflex over ( )}2, and the camera ismounted more than 5 mm away from the head. The sensor measures a signalindicative of a respiration rate of the user. The computer receivesprevious images of the area, which are indicative of a previous facialblood flow pattern while the user had a certain extent of CHF, and aprevious respiration rate taken while the user had the certain extent ofCHF. The computer identifies exacerbation of the CHF based on: adifference above a first predetermined threshold between the facialblood flow pattern and the previous facial blood flow pattern, and anincrease above a second predetermined threshold in the respiration ratecompared to the previous respiration rate.

Yet another aspect of this disclosure involves a system configured tocalculate extent of congestive heart failure (CHF). In some embodiments,the system includes smartglasses configured to be worn on a user's head,an inward-facing camera, which is physically coupled to thesmartglasses, and a computer. The inward-facing camera captures imagesof an area comprising skin on the user's head. Optionally, the area islarger than 4 cm{circumflex over ( )}2, and the camera is mounted morethan 5 mm away from the head. The computer receives a first set of theimages taken while the user was at rest and prior to a period duringwhich the user performed physical activity, and a second set of theimages taken within ten minutes after the period. The computer thencalculates the extent of CHF based on differences in facial blood flowpatterns recognizable in the first and second sets of the images.

In one embodiment, the system includes a movement sensor, physicallycoupled to the smartglasses, which measures movements of the user. Inthis embodiment, the computer detects the period during which the userperformed the physical activity based on the movements. Optionally, thephysical activity involves walking at least 20 steps.

In another embodiment, the computer calculate first and second series ofheart rate values from portions of iPPG signals extracted from the firstand second sets of images, respectively. In this embodiment, thecomputer calculates the extent of the CHF also based on the extent towhich heart rate values in the second series were above heart ratevalues in the first series. Optionally, the computer calculates theextent of CHF based on a duration after the period in which the heartrate values in the second series were above the heart rate values in thefirst series.

In yet another embodiment, the computer includes a head-mounted sensorthat measures temperature of a region comprising skin on the user's head(T_(skin)). In this embodiment, the computer utilizes T_(skin) tocompensate for effects of skin temperature on the facial blood flowpattern.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments are herein described by way of example only, withreference to the following drawings:

FIG. 1a is a schematic illustration of components of a system configuredto calculate an extent of CHF and/or identify exacerbation of CHF;

FIG. 1b illustrates various possibilities for positioning one or moreinward-facing cameras on smartglasses;

FIG. 1c illustrates an embodiment of smartglasses that are part of asystem for calculating an extent of CHF that a user is suffering from,and/or identify exacerbation of CHF;

FIG. 2 is a schematic illustration of some of the various fiducialpoints for PPG signals often used in the art;

FIG. 3a illustrates smartglasses which include contactphotoplethysmographic devices;

FIG. 3b illustrates smartglasses that include first and secondinward-facing cameras;

FIG. 4 illustrates smartglasses that include four inward-facing cameras;

FIG. 5 illustrates an embodiment of a system that includes a singlehead-mounted thermal camera that may be used to detect a stroke;

FIG. 6 illustrates a scenario in which an alert regarding a possiblestroke is issued;

FIG. 7, FIG. 8, FIG. 9, and FIG. 10 illustrate physiological andbehavioral changes that may occur following a stroke;

FIG. 11, FIG. 12, FIG. 13, and FIG. 14 illustrates various activitiesthat a person may be requested to perform in order to determine whetherthe person has suffered from a stroke;

FIG. 15 and FIG. 16 illustrate the difference between a timelyintervention in the event of a stroke and intervention that comes toolate;

FIG. 17a illustrates one embodiment of a system that includes multiplepairs of right and left cameras and locations on the face that they maybe used to measure;

FIG. 17b illustrates a stroke sign that involves decreased blood flow inthe forehead;

FIG. 17c illustrates a stroke sign that involves decreased blood flow ina periorbital region;

FIG. 18a and FIG. 18b illustrate embodiments of a system with ahead-mounted camera located behind the ear;

FIG. 18c illustrates an ischemic stroke that restricts the blood flow tothe side of the head, which may be detected by embodiments describedherein;

FIG. 19, FIG. 20, FIG. 21, and FIG. 22 illustrate head-mounted systems(HMSs) configured to measure various ROIs relevant to some of theembodiments describes herein; and

FIG. 23a and FIG. 23b are schematic illustrations of possibleembodiments for computers.

DETAILED DESCRIPTION

Herein the terms “photoplethysmogram signal”, “photoplethysmographicsignal”, “photoplethysmography signal”, and other similar variations areinterchangeable and refer to the same type of signal. Aphotoplethysmogram signal may be referred to as a “PPG signal”, or an“iPPG signal” when specifically referring to a PPG signal obtained froma camera. The terms “photoplethysmography device”,“photoplethysmographic device”, “photoplethysmogram device”, and othersimilar variations are also interchangeable and refer to the same typeof device that measures a signal from which it is possible to extractthe photoplethysmogram signal. The photoplethysmography device may bereferred to as “PPG device”.

Sentences in the form of “a sensor configured to measure a signalindicative of a photoplethysmogram signal” refer to at least one of: (i)a contact PPG device, such as a pulse oximeter that illuminates the skinand measures changes in light absorption, where the changes in lightabsorption are indicative of the PPG signal, and (ii) a non-contactcamera that captures images of the skin, where a computer extracts thePPG signal from the images using an imaging photoplethysmography (iPPG)technique. Other names known in the art for iPPG include: remotephotoplethysmography (rPPG), remote photoplethysmographic imaging,remote imaging photoplethysmography, remote-PPG, and multi-sitephotoplethysmography (MPPG). Additional names known in the art for iPPGfrom the face include: facial hemoglobin concentration changes, dynamichemoglobin concentration/information extraction, facial blood flowchanges, and transdermal optical imaging.

A PPG signal is often obtained by using a pulse oximeter, whichilluminates the skin and measures changes in light absorption. Anotherpossibility for obtaining the PPG signal is using an imagingphotoplethysmography (iPPG) device. As opposed to contact PPG devices,iPPG does not require contact with the skin and is obtained by anon-contact sensor, such as a video camera.

A time series of values measured by a PPG device, which is indicative ofblood flow changes due to pulse waves, is typically referred to as awaveform (or PPG waveform to indicate it is obtained with a PPG device).It is well known that PPG waveforms show significant gender-relateddifferences, age-related differences, and health-related differences. Asa result, the PPG waveforms of different people often display differentcharacteristics (e.g., slightly different shapes and/or amplitudes). Inaddition, the PPG waveform depends on the site at which it is measured,skin temperature, skin tone, and other parameters.

The analysis of PPG signals usually includes the following steps:filtration of a PPG signal (such as applying bandpass filtering and/orheuristic filtering), extraction of feature values from fiducial pointsin the PPG signal (and in some cases may also include extraction offeature values from non-fiducial points in the PPG signal), and analysisof the feature values.

One type of features that is often used when performing calculationsinvolving PPG signals involves fiducial points related to the waveformsof the PPG signal and/or to functions thereof (such as variousderivatives of the PPG signal). There are many known techniques toidentify the fiducial points in the PPG signal, and to extract thefeature values. The following are some non-limiting examples of how toidentify fiducial points.

FIG. 2 is a schematic illustration some of the various fiducial pointsoften used in the art (and described below). These examples of fiducialpoints include fiducial points of the PPG signal, fiducial points in thefirst derivative of the PPG signal (velocity photoplethysmogram, VPG),and fiducial points in the second derivative of the PPG signal(acceleration photoplethysmogram, APG).

Fiducial points in the PPG signal may include: the systolic notch 120,which is the minimum at the PPG signal onset; the systolic peak 121,which is the maximum of the PPG signal; the dicrotic notch 122, whichcoincident with e 134 (see below at the second derivative of the PPGsignal); and the diastolic peak 123, which is the first local maximum ofthe PPG signal after the dicrotic notch and before 0.8 of the durationof the cardiac cycle, or if there is no such local maximum, then thefirst local maximum of the second derivative after e and before 0.8 ofthe duration of the cardiac cycle.

Fiducial points in the first derivative of the PPG signal (velocityphotoplethysmogram, VPG) may include: the maximum slope peak in systolicof VPG 125; the local minima slope in systolic of VPG 126; the globalminima slope in systolic of VPG 127; and the maximum slope peak indiastolic of VPG 128.

Fiducial points in the second derivative of the PPG signal (accelerationphotoplethysmogram, APG) may include: a 130, which is the maximum of APGprior to the maximum of VPG; b 131, which is the first local minimum ofAPG following a; c 132, which is the greatest maximum of APG between band e, or if no maxima then the first of (i) the first maximum of VPGafter e, and (ii) the first minimum of APG after e; d 133, which is thelowest minimum of APG after c and before e, or if no minima thencoincident with c; e 134, which is the second maximum of APG aftermaximum of VPG and before 0.6 of the duration of the cardiac cycle,unless the c wave is an inflection point, in which case take the firstmaximum; and f 135, which is the first local minimum of APG after e andbefore 0.8 of the duration of the cardiac cycle.

Fiducial points in the third derivative of the PPG signal (PPG′″) mayinclude: the first local maximum of PPG′″ after b; and the last localminimum of PPG′″ before d, unless c=d, in which case take the firstlocal minimum of PPG′″ after d, and if there is a local maximum of thePPG signal between this point and the dicrotic notch then use itinstead.

Feature values of the PPG signal may also be extracted fromrelationships in the PPG signal and/or its derivatives. The followingare some non-limiting examples such possible feature values: pulsewidth, peak to peak time, ratio of areas before and after dicrotic notchin a complete cycle, baseline wander (BW), which is the mean of theamplitudes of a beat's peak and trough; amplitude modulation (AM), whichis the difference between the amplitudes of each beat's peak and trough;and frequency modulation (FM), which is the time interval betweenconsecutive peaks.

Examples of additional features that can be extracted from the PPGsignal, together with schematic illustrations of the feature locationson the PPG signal, can be found in the following three publications: (i)Peltokangas, Mikko, et al. “Parameters extracted from arterial pulsewaves as markers of atherosclerotic changes: performance andrepeatability.” IEEE journal of biomedical and health informatics 22.3(2017): 750-757; (ii) Ahn, Jae Mok. “New aging index using signalfeatures of both photoplethysmograms and acceleration plethysmograms.”Healthcare informatics research 23.1 (2017): 53-59; (iii) Charlton,Peter H., et al. “Assessing mental stress from the photoplethysmogram: anumerical study.” Physiological measurement 39.5 (2018): 054001, and(iv) Peralta, Elena, et al. “Optimal fiducial points for pulse ratevariability analysis from forehead and finger photoplethysmographicsignals.” Physiological measurement 40.2 (2019): 025007.

Although the above mentioned references describe manual featureselection, the features may be selected using any appropriate featureengineering technique, including using automated feature engineeringtools that help data scientists to reduce data exploration time, andenable non-experts, who may not be familiar with data science and/or PPGcharacteristics, to quickly extract value from their data with littleeffort.

Unless there is a specific reference to a specific derivative of the PPGsignal, phrases of the form of “based on the PPG signal” refer to thePPG signal and any derivative thereof, including the first derivative ofthe PPG signal, the second derivative of the PPG signal, and the thirdderivative of the PPG signal. For example, a sentence in the form of “acomputer configured to detect a physiological signal based on the PPGsignal” is to be interpreted as “a computer configured to detect aphysiological signal based on at least one of: the PPG signal, a firstderivative of the PPG signal, a second derivative of the PPG signal, athe third derivative of the PPG signal, and/or any other derivative ofthe PPG signal”.

Algorithms for filtration of the PPG signal (and/or the images in thecase of iPPG), extraction of feature values from fiducial points in thePPG signal, and analysis of the feature values extracted from the PPGsignal are well known in the art, and can be found for example in thefollowing references: (i) Allen, John. “Photoplethysmography and itsapplication in clinical physiological measurement.” Physiologicalmeasurement 28.3 (2007): R1, and also in the thousands of referencesciting this reference; (ii) Elgendi, Mohamed. “On the analysis offingertip photoplethysmogram signals.” Current cardiology reviews 8.1(2012): 14-25, and also in the hundreds of references citing thisreference; (iii) Holton, Benjamin D., et al. “Signal recovery in imagingphotoplethysmography.” Physiological measurement 34.11 (2013): 1499, andalso in the dozens of references citing this reference, (iv) Sun, Yu,and Nitish Thakor. “Photoplethysmography revisited: from contact tononcontact, from point to imaging.” IEEE Transactions on BiomedicalEngineering 63.3 (2015): 463-477, and also in the dozens of referencesciting this reference, (v) Kumar, Mayank, Ashok Veeraraghavan, andAshutosh Sabharwal. “DistancePPG: Robust non-contact vital signsmonitoring using a camera.” Biomedical optics express 6.5 (2015):1565-1588, and also in the dozens of references citing this reference,(vi) Wang, Wenjin, et al. “Algorithmic principles of remote PPG.” IEEETransactions on Biomedical Engineering 64.7 (2016): 1479-1491, and alsoin the dozens of references citing this reference, and (vii) Rouast,Philipp V., et al. “Remote heart rate measurement using low-cost RGBface video: a technical literature review.” Frontiers of ComputerScience 12.5 (2018): 858-872, and also in the dozens of referencesciting this reference.

In the case of iPPG, the input comprises images having multiple pixels.The images from which the iPPG signal and/or the hemoglobinconcentration patterns are extracted may undergo various preprocessingto improve the signal, such as color space transformation, blind sourceseparation using algorithms such as independent component analysis (ICA)or principal component analysis (PCA), and various filtering techniques,such as detrending, bandpass filtering, and/or continuous wavelettransform (CWT). Various preprocessing techniques known in the art thatmay assist in extracting iPPG signals from images are discussed inZaunseder et al. (2018), “Cardiovascular assessment by imagingphotoplethysmography—a review”, Biomedical Engineering 63(5), 617-634.

Various embodiments described herein involve calculations based onmachine learning approaches. Herein, the terms “machine learningapproach” and/or “machine learning-based approaches” refer to learningfrom examples using one or more approaches. Examples of machine learningapproaches include: decision tree learning, association rule learning,regression models, nearest neighbors classifiers, artificial neuralnetworks, deep learning, inductive logic programming, support vectormachines, clustering, Bayesian networks, reinforcement learning,representation learning, similarity and metric learning, sparsedictionary learning, genetic algorithms, rule-based machine learning,and/or learning classifier systems.

Herein, a “machine learning-based model” is a model trained using one ormore machine learning approaches. For brevity's sake, at times, a“machine learning-based model” may simply be called a “model”. Referringto a model as being “machine learning-based” is intended to indicatethat the model is trained using one or more machine learning approaches(otherwise, “model” may also refer to a model generated by methods otherthan machine learning).

Herein, “feature values” (also known as feature vector, feature data,and numerical features) may be considered input to a computer thatutilizes a model to perform the calculation of a value, such as a valueindicative of one or more vital signs of a user. It is to be noted thatthe terms “feature” and “feature value” may be used interchangeably whenthe context of their use is clear. However, a “feature” typically refersto a certain type of value, and represents a property, while “featurevalue” is the value of the property with a certain instance (i.e., thevalue of the feature in a certain sample).

It is to be noted that when it is stated that feature values aregenerated based on data comprising multiple sources, it means that foreach source, there is at least one feature value that is generated basedon that source (and possibly other data). For example, stating thatfeature values are generated from an image capturing first and secondregions (IM_(ROI1) and IM_(ROI2), respectively) means that the featurevalues include at least a first feature value generated based onIM_(ROI1) and a second feature value generated based on IM_(ROI2).

In addition to feature values generated based on measurements taken bysensors mentioned in a specific embodiment, at least some feature valuesutilized by a computer of the specific embodiment may be generated basedon additional sources of data that were not specifically mentioned inthe specific embodiment. Some examples of such additional sources ofdata include: (i) contextual information such as the time of day (e.g.,to account for effects of the circadian rhythm), day of month (e.g., toaccount for effects of the lunar rhythm), day in the year (e.g., toaccount for seasonal effects), and/or stage in a menstrual cycle; (ii)information about the user being measured such as sex, age, weight,height, body build, genetics, medical records, and/or intake ofsubstances; (iii) measurements of the environment, such as temperature,humidity level, noise level, elevation, air quality, a wind speed,precipitation, and infrared radiation; and/or (iv) values ofphysiological signals of the user obtained by sensors that are notmentioned in the specific embodiment, such as an electrocardiogram (ECG)sensor, an electroencephalography (EEG) sensor, a galvanic skin response(GSR) sensor, a movement sensor, an acoustic sensor, and/or atemperature sensor.

A machine learning-based model of a specific embodiment may be trained,in some embodiments, based on data collected in day-to-day, real worldscenarios. As such, the data may be collected at different times of theday, while users perform various activities, and in variousenvironmental conditions. Utilizing such diverse training data mayenable a trained model to be more resilient to the various effects thatdifferent conditions can have on the measurements, and consequently, beable to achieve better detection of a required parameter in real worldday-to-day scenarios.

The machine learning-based model may be personalized for a specificuser. For example, after receiving a verified diagnosis of an extent ofa physiological condition (such as blood pressure level, extent of acardiovascular disease, extent of a pulmonary disease, extent of amigraine attack, etc.), the computed can use the verified diagnosis aslabels and generate from a physiological measurement (such as the PPGsignal, the temperature signal, the movement signal, and/or the audiosignal) feature values to train a personalized machine learning-basedmodel for the user. Then the computer can utilize the personalizedmachine learning-based model for future calculations of the extent ofthe physiological condition based on feature values.

Sentences in the form of “inward-facing head-mounted camera” refer to acamera configured to be worn on a user's head and to remain pointed atits ROI, which is on the user's face, also when the user's head makesangular and lateral movements (such as movements with an angularvelocity above 0.1 rad/sec, above 0.5 rad/sec, and/or above 1 rad/sec).A head-mounted camera (which may be inward-facing and/or outward-facing)may be physically coupled to a frame worn on the user's head, may bephysically coupled to eyeglasses using a clip-on mechanism (configuredto be attached to and detached from the eyeglasses), may be physicallycoupled to a hat or a helmet, or may be mounted to the user's head usingany other known device that keeps the camera in a fixed positionrelative to the user's head also when the head moves. Sentences in theform of “sensor physically coupled to the frame” mean that the sensormoves with the frame, such as when the sensor is fixed to (or integratedinto) the frame, and/or when the sensor is fixed to (or integrated into)an element that is physically coupled to the frame, and/or when thesensor is connected to the frame with a clip-on mechanism.

Sentences in the form of “a frame configured to be worn on a user'shead” or “a frame worn on a user's head” refer to a mechanical structurethat loads more than 50% of its weight on the user's head. For example,an eyeglasses frame may include two temples connected to two rimsconnected by a bridge; the frame in Oculus Rift™ includes the foamplaced on the user's face and the straps; and the frame in Google Glass™is similar to an eyeglasses frame. Additionally or alternatively, theframe may connect to, be affixed within, and/or be integrated with, ahelmet (e.g., a safety helmet, a motorcycle helmet, a combat helmet, asports helmet, a bicycle helmet, etc.), goggles, and/or abrainwave-measuring headset.

Sentences in the form of “a frame configured to be worn on a user's headin a consistent manner” refer to a frame that is located in the sameposition relative to the head when worn repeatedly, and thus sensorsattached to that frame are most likely to be positioned each time at thesame location relative to the head. For example, eyeglasses frames,goggles, and helmets are all included under the definition of a framethat is worn in a consistent manner. However, a flexible headband, oradhesive sensors that are placed manually one by one, are not worn in aconsistent manner, because these sensors are most likely to bepositioned each time in a different location relative to the head.

The term “smartglasses” refers to any type of a device that resembleseyeglasses, and includes a frame configured to be worn on a user's headin a consistent manner, and includes electronics to operate one or moresensors. The frame may be an integral part of the smartglasses, and/oran element that is connected to the smartglasses. Examples ofsmartglasses include: any type of eyeglasses with electronics (whetherprescription or plano), sunglasses with electronics, safety goggles withelectronics, sports goggle with electronics, augmented reality devices,virtual reality devices, and mixed reality devices. In addition, theterm “eyeglasses frame” refers to one or more of the following devices,whether with or without electronics: smartglasses, prescriptioneyeglasses, plano eyeglasses, prescription sunglasses, plano sunglasses,safety goggles, sports goggle, an augmented reality device, virtualreality devices, and a mixed reality device.

The term “smart-helmet” refers to a helmet that includes a frameconfigured to be worn on a user's head in a consistent manner, andincludes electronics to operate one or more sensors. The frame may be anintegral part of the smart-helmet, and/or an element that is connectedto the smart-helmet. Examples of smart-helmets include: a safety helmetwith electronics, a motorcycle helmet with electronics, a combat helmetwith electronics, a sports helmet with electronics, and a bicycle helmetwith electronics.

Examples of electronics that may be included in smartglasses and/or asmart-helmet include one or more of the following electronic components:a computer, a microcontroller, a processor, a memory, and acommunication interface. The electronics of the smartglasses and/orsmart-helmets may be integrated in various ways. For example, theelectronics may be integrated into the package of one of the sensors,such as a camera housing that is physically coupled to a helmet, wherethe housing includes the imaging sensor and its processor, memory, powersupply and wireless communication unit. In another example, theelectronics may be integrated into the frame, such as a microcontroller,power supply and wireless communication unit that are integrated into aneyeglasses frame, and configured to operate a PPG device and amicrophone that are physically coupled to the frame.

The term “Visible-light camera” refers to a non-contact device designedto detect at least some of the visible spectrum, such as a video camerawith optical lenses and CMOS or CCD sensor. The term “thermal camera”refers to a non-contact device that measures electromagnetic radiationhaving wavelengths longer than 2500 nanometer (nm) and does not touchits region of interest (ROI). A thermal camera may include one sensingelement (pixel), or multiple sensing elements that are also referred toherein as “sensing pixels”, “pixels”, and/or focal-plane array (FPA). Athermal camera may be based on an uncooled thermal sensor, such as athermopile sensor, a microbolometer sensor (where microbolometer refersto any type of a bolometer sensor and its equivalents), a pyroelectricsensor, or a ferroelectric sensor.

A reference to a “camera” herein may relate to various types of devices.In one example, a camera may be a visible-light camera. In anotherexample, a camera may capture light in the ultra-violet range. Inanother example, a camera may capture near infrared radiation (e.g.,wavelengths between 750 and 2000 nm). And in still another example, acamera may be a thermal camera.

When a camera is inward-facing and head-mounted, challenges faced bysystems known in the art that are used to acquire images, which includenon-head-mounted cameras, may be simplified and even eliminated withsome of the embodiments described herein. Some of these challenges mayinvolve dealing with complications caused by movements of the user,image registration, region of interest (ROI) alignment, tracking basedon hot spots or markers, and motion compensation.

The term “temperature sensor” refers to a device that measurestemperature and/or temperature change. The temperature sensor may be acontact thermometer (such as a thermistor, a thermocouple), and/or anon-contact thermal cameras (such as a thermopile sensor, amicrobolometer sensor, a pyroelectric sensor, or a ferroelectricsensor). Some examples of temperature sensors useful to measure skintemperature include: thermistors, thermocouples, thermoelectic effect,thermopiles, microbolometers, and pyroelectric sensors. Some examples oftemperature sensors useful to measure environment temperature include:thermistors, resistance temperature detectors, thermocouples;thermopiles, and semiconductor-based sensors.

The term “movement sensor” refers to a sensor comprising one or more ofthe following components: a 3-axis gyroscope, a 3-axis accelerometer,and a 3-axis magnetometer. The movement sensor may also include a sensorthat measures barometric pressure.

The term “acoustic sensor” refers to a device that converts sound wavesinto an electrical signal. An acoustic sensor can be a microphone, suchas a dynamic microphone that works via electromagnetic induction, apiezoelectric microphone that uses the phenomenon of piezoelectricity, afiber-optic microphone that converts acoustic waves into electricalsignals by sensing changes in light intensity, aMicro-Electrical-Mechanical System (MEMS) microphone (such as siliconMEMS and piezoelectric MEMS), and/or other sensors that measure soundwaves, such as described in the following examples: (i) Han, Jae Hyun,et al. “Basilar membrane-inspired self-powered acoustic sensor enabledby highly sensitive multi tunable frequency band.” Nano Energy 53(2018): 198-205, describes a self-powered flexible piezoelectricacoustic sensor having high sensitivity, (ii) Rao, Jihong, et al.“Recent Progress in Self-Powered Skin Sensors.” Sensors 19.12 (2019):2763. describes various self-powered acoustic skin sensors, such as anintegrated triboelectric nanogenerator (TENG) with a polymer tube thatcan pick up and recover human throat voice even in an extremely noisy orwindy environment, and (iii) Scanlon, Michael V. Acoustic sensor forvoice with embedded physiology. Army Research Lab Adelphi Md., 1999,describes a gel-coupled acoustic sensor able to collect informationrelated to the function of the heart, lungs, and changes in voicepatterns.

Herein, the term “blood pressure” is indicative of one or more of thefollowing: the systolic blood pressure of the user, the diastolic bloodpressure of the user, and the mean arterial pressure (MAP) of the user.It is specifically noted that the term “blood pressure” is not limitedto the systolic and diastolic blood pressure pair.

The terms “substance intake” or “intake of substances” refer to any typeof food, beverage, medications, drugs, smoking/inhaling, and anycombination thereof.

The following is a description of embodiments of systems for calculatingan extent of congestive heart failure (CHF) that a user is sufferingfrom, and/or identify exacerbation of CHF. The systems rely on measuringsigns of CHF that include changes to facial blood flow, respiration,and/or level of activity. Embodiments of these systems include:smartglasses with various sensors and/or cameras coupled thereto, and acomputer that is used to calculate the extent of CHF and/or to detectthe exacerbation of CHF.

FIG. 1a is a schematic illustration of components of a system configuredto calculate an extent of CHF and/or identify exacerbation of CHF. Inone embodiment, the system includes at least a pair of smartglasses(e.g., smartglasses 800 or smartglasses 805, illustrated in FIG. 1b andFIG. 1c , respectively), and an inward-facing camera 820, such a camerafrom among cameras 802 a, 802 b, and 802 c illustrated in FIG. 1b , or acamera from among cameras 806 a and 806 b illustrated in FIG. 1c . Thesystem also includes computer 828. The system may include additionalelements such as a sensor 822, which is configured to measure a signalindicative of respiration, a movement sensor, such as inertialmeasurement unit (IMU) 830, a skin temperature sensor 824, anenvironment temperature sensor 826, and/or a user interface 832.

The smartglasses are configured to be worn on a user's head. Optionally,various sensors and/or cameras that are physically coupled to thesmartglasses, e.g., by being attached to and/or embedded in the frame ofthe smartglasses, are used to measure the user while the user wears thesmartglasses. Optionally, at least some of the sensors and/or camerasthat are physically coupled to the smartglasses may be utilized tomeasure the environment in which the user is in. In one example, thesmartglasses are eyeglasses with sensors and electronics attachedthereto and/or embedded therein. In another example, the smartglassesmay be an extended reality device (i.e., an augmented realty device, avirtual reality device, and/or mixed reality device).

FIG. 1c illustrates an embodiment of smartglasses that are part of asystem for calculating an extent of congestive heart failure (CHF) thata user is suffering from, and/or identify exacerbation of CHF. Thesystem includes smartglasses 805, which have various components coupledthereto. These components include the inward-facing cameras 806 a and806 b, which capture images that include portions of the left and rightsides of the forehead, respectively. Additionally, the smartglasses 805may include a microphone 809, which can be used to measure a signalindicative of respiration, a temperature sensor 811 that measurestemperature on the user's face, a temperature sensor 812 that measurestemperature of the environment, and IMU 814. The smartglasses 805 alsoinclude computer 828, which in this embodiment is physically coupled tothe frame of the smartglasses 805.

One or more inward-facing cameras, such as the inward-facing camera 820,are physically coupled to the smartglasses. Each of these cameras isconfigured to capture images of an area comprising skin on the user'shead. For example, the inward-facing camera 820 captures images 821 ofan area on the user's head. Optionally, the area comprises a portion ofthe lips of the user. Additional examples of the area are illustrated inFIG. 1b , which describes various possibilities for positioning one ormore inward-facing cameras on smartglasses 800. Inward-facing camera 802a is located above the nose bridge and captures images of an area 803 aon the user's forehead. Inward-facing cameras 802 b and 802 c arelocated on the left and right sides of the smartglasses 800,respectively; they capture images that include areas 803 b and 803 c onthe left and right sides of the user's face, respectively.

Inward-facing cameras, such as the inward-facing camera 820, areoptionally mounted more than 5 mm away from the head and/or at least 5mm away from the area appearing in the images they capture. Optionally,the inward-facing camera 820 is mounted less than 10 cm away from thehead (i.e., the frame of the smartglasses holds the camera at a distancethat is smaller than 10 cm from the head). Optionally, each areaappearing in images captured by an inward-facing camera, such as theinward-facing camera 820, is larger than 4 cm{circumflex over ( )}2. Insome embodiments, the area on the user's head appearing in images 821 isnot occluded by the inward-facing camera 820.

Cameras utilized in embodiments described herein (including theinward-facing camera 820) are typically small and lightweight. In someembodiments, each camera weighs below 10 g and even below 2 g. In oneexample, the inward-facing camera 820 camera is a multi-pixel videocamera having a CMOS or a CCD sensor. The video camera may captureimages at various rates. In one example, the images 821 are captured ata frame rate of at least 30 frames per second (fps). In another example,the images 821 are captured at a frame rate of at least 100 fps. Instill another example, the images 821 are captured at a frame rate of atleast 256 fps. Images taken by inward-facing cameras may have variousresolutions. In one example, the images 821 have a resolution of atleast 8×8 pixels. In another example, the images 821 have a resolutionof at least 32×32 pixels. In yet another example, the images 821 have aresolution of at least 640×480 pixels.

In some embodiments, at least one of the inward-facing cameras maycapture light in the near infrared spectrum (NIR). Optionally, each ofthe at least one of the inward-facing cameras may include optics andsensors that capture light rays in at least one of the following NIRspectrum intervals: 700-800 nm, 700-900 nm, 700-1,000 nm. Optionally,the sensors may be CCD sensors designed to be sensitive in the NIRspectrum and/or CMOS sensors designed to be sensitive in the NIRspectrum.

In some embodiments, the system may include an optical emitterconfigured to direct electromagnetic radiation at an area on the user'shead that appears in images captured by an inward-facing camera.Optionally, the optical emitter comprises one or more of the following:a laser diode (LD), a light-emitting diodes (LED), and an organiclight-emitting diode (OLED).

It is to be noted that when embodiments described in this disclosureutilize optical emitters directed at a region of interest (ROI), such asan area appearing in images captured by an inward-facing camera, theoptical emitter may be positioned in various locations relative to theROI. In some embodiments, the optical emitter may be positionedessentially directly above the ROI, such that electromagnetic radiationis emitted at an angle that is perpendicular (or within 10 degrees frombeing perpendicular) relative to the ROI. Optionally, an inward-facingcamera may be positioned near the optical emitter in order to capturethe reflection of electromagnetic radiation from the ROI. In otherembodiments, the optical emitter may be positioned such that it is notperpendicular to the ROI. Optionally, the optical emitter does notocclude the ROI. In one example, the optical emitter may be located atthe top of a frame of a pair of eyeglasses, and the ROI may include aportion of the forehead. In another example, optical emitter may belocated on an arm of a frame of a pair of eyeglasses and the ROI may belocated above the arm or below it.

The computer 828 may utilize various preprocessing approaches in orderto assist in calculations and/or in extraction of an iPPG signal fromthe images 821. Optionally, the images 821 may undergo variouspreprocessing steps prior to being used by the computer 828. Somenon-limiting examples of the preprocessing include: normalization ofpixel intensities (e.g., to obtain a zero-mean unit variance time seriessignal), and conditioning a time series signal by constructing a squarewave, a sine wave, or a user defined shape, such as that obtained froman ECG signal or a PPG signal as described in U.S. Pat. No. 8,617,081.Additionally or alternatively, images may undergo various preprocessingto improve the signal, such as color space transformation (e.g.,transforming RGB images into a monochromatic color or images in adifferent color space), blind source separation using algorithms such asindependent component analysis (ICA) or principal component analysis(PCA), and various filtering techniques, such as detrending, bandpassfiltering, and/or continuous wavelet transform (CWT). Variouspreprocessing techniques known in the art that may assist in extractingan iPPG signal from the images 821 are discussed in Zaunseder et al.(2018), “Cardiovascular assessment by imaging photoplethysmographyareview”, Biomedical Engineering 63(5), 617-634. An example ofpreprocessing that may be used in some embodiments is given in U.S. Pat.No. 9,020,185, titled “Systems and methods for non-contact heart ratesensing”, which describes how times-series signals obtained from videoof a user can be filtered and processed to separate an underlyingpulsing signal by, for example, using an ICA algorithm.

Due to the proximity of the one or more inward-facing cameras to theface, in some embodiments, there may be an acute angle between theoptical axis of an inward-facing camera and the area captured by imagestaken by said camera (e.g., when the area is on, and/or includes aportion of, the forehead or a cheek). In order to improve the sharpnessof images captured by said camera, camera may be configured to operatein a way that takes advantage of the Scheimpflug principle. In oneembodiment, camera includes a sensor and a lens; the sensor plane istilted by a fixed angle greater than 2° relative to the lens planeaccording to the Scheimpflug principle in order to capture a sharperimage when the smartglasses are worn by the user (where the lens planerefers to a plane that is perpendicular to the optical axis of the lens,which may include one or more lenses). In another embodiment, the cameraincludes a sensor, a lens, and a motor; the motor tilts the lensrelative to the sensor according to the Scheimpflug principle. The tiltimproves the sharpness of images when the smartglasses are worn by theuser. Additional details regarding the application of the Scheimpflugprinciple are discussed further below.

Variations in the reflected ambient light may introduce artifacts intoimages collected with inward-facing cameras that can add noise to theseimages and make detections and/or calculations based on these imagesless accurate. In some embodiments, the system includes anoutward-facing camera, which is coupled to the smartglasses, and takesimages of the environment. Optionally, this outward-facing camera islocated less than 10 cm from the user's face and weighs below 10 g.Optionally, the outward-facing camera may include optics that provide itwith a wide field of view.

Changes in respiration pattern (in particular an increased respirationrate) are often associated with CHF and its exacerbation. In order toincorporate information about respiration into detection of CHF, someembodiments include the sensor 822, which is physically coupled to thesmartglasses, and configured to measure a respiration signal 823, whichis a signal indicative of the user's respiration rate. Various types ofsensors may be utilized for this purpose.

In one embodiment, the sensor 822 may be a microphone from among one ormore microphones coupled to the smartglasses. The computer 828 mayutilize the microphone to measure audio signal comprising sounds in theuser's vicinity, and the respiration signal 823 is derived from thesounds. From this respiration signal, values of respiration parametersmay be extracted (such as the respiration rate). Various algorithmicapproaches may be utilized to extract parameters related to respirationfrom an acoustic signal. Some examples of possible approaches areprovided in Avalur, D. S. “Human breath detection using a microphone”,Diss. Faculty of Science and Engineering, 2013. Additional algorithmicapproaches that may be utilized, in some embodiments, are described inU.S. Pat. No. 7,850,619, titled “Apparatus and method for breathingpattern determination using a non-contact microphone”, and in US patentapplication No. 2019/0029563, titled “Methods and apparatus fordetecting breathing patterns”.

In another embodiment, the sensor 822 may be an inward-facinghead-mounted thermal camera and the respiration signal 823 may include,and/or be derived from, thermal measurements of a region below thenostrils (TH_(ROI)) of a user, where TH_(ROI) are indicative of theexhale stream. Optionally, the computer 828 generates certain featurevalues based on TH_(ROI), and utilizes a certain model to calculate arespiratory parameter, such as the respiration rate, based on thecertain feature values. Optionally, the certain model is trained basedon previous TH_(ROI) of the user. Additional details regardingutilization of head-mounted thermal cameras to measure signalsindicative of respiratory activity are provided in U.S. Pat. No.10,130,308, titled “Calculating respiratory parameters from thermalmeasurements”, and U.S. Pat. No. 10,136,856, titled “Wearablerespiration measurements system”.

The computer 828 is configured, in some embodiments, to calculate theextent of CHF based on facial blood flow patterns recognizable theimages 821 taken by the inward-facing camera 820 and optionallyadditional inward-facing cameras. Optionally, the computer 828 mayutilize additional information such as a respiration rate recognizablein the respiration signal 823. Examples of computers that may beutilized to perform this calculation are computer 400 or computer 410illustrated in FIG. 23a and FIG. 23b , respectively.

In different embodiments, the computer 828 may refer to differentcomponents and/or a combination of components. In some embodiments, thecomputer 828 may include a processor located on the smartglasses (asillustrated in FIG. 1c ). In other embodiments, at least some of thecalculations attributed to the computer 828 may be performed on a remoteprocessor, such as a cloud-based server. Thus, references tocalculations being performed by the “computer 828” should be interpretedas calculations being performed utilizing one or more computers, withsome of these one or more computers possibly being remote of thesmartglasses. For example, some of the operations attributed to thecomputer 828 may be performed, in some embodiment's, by a processor inanother device of the user (e.g., a processor on a smartphone) or by acloud-based server.

The images 821 may provide values of coloration intensities (i.e.,intensities detected at one or more light wavelengths) at differentportions of the area on the user's head, which correspond to thedifferent pixels in the images. In some embodiments, the computer 828may utilize various computational techniques described herein to extracta photoplethysmogram signal (iPPG signal) from the images 821. In someembodiments, the computer 828 may extract from the images 821 colorationvalues that do not exhibit a temporal variation that corresponds to theuser's cardiac pulses (e.g., average color values for various pixelsduring a period spanning a duration of several seconds, or longer).

The coloration intensities may represent a facial blood flow patternthat is recognizable in the images 821. A facial blood flow patternrepresents characteristics of blood flow in the area captured in theimages 821, while the images 821 were taken. A “facial blood flowpattern” may refer to various types of patterns, in embodimentsdescribed herein. The following are some examples of types of patternsthat may be recognizable in the images 821.

In some embodiments, a facial blood flow pattern refers to a colormapping of various portions of the area captured in the images 821(e.g., the mapping provides the colors of different pixels in the images821). Optionally, the mapping provides average intensities of one ormore colors of the pixels over a period of time during which the images821 were taken. Since the extent of cutaneous blood flow affects theskin coloration, the colors observed in the images 821 can indicate theextent of blood flow in different portions of the area. For example,increased blood flow can be detected as higher pixel intensities for thecolor red (due to the flushness accompanying the increase blood flow).In another example, CHF may lead to decreased blood flow, which maycause colors in the area to change towards pallor (an extremely pale orgrayish coloring of the face) or change due to cyanosis (having a bluishcoloring of the nails, lips and possibly the skin). The occurrence ofCHF can lead to slight changes in the colors, which may not be detectedby the naked eye. However, the head-mounted system can detect theseslight (or distinct) color changes, and track their trend that isindicative of the extent of CHF.

In other embodiments, a facial blood flow pattern may refer to timeseries data, such as a sequence of images representing a progression ofa pulse wave in the area. Different extents of blood flow may producedifferent sequences of representative images, which depend on thestructure of the facial blood vessels of the user. For example, for thesame user and area, a stronger blood flow can be represented by asequence of images with larger color variations than the color variationin a sequence of images representing a weaker blood flow in the area.

In still other embodiments, a facial blood flow pattern may refer to acontour map, representing the extent to which pixels at a certainwavelength (e.g., corresponding to the color red) have at least acertain value. Since the extent of blood flow is correlated with anincrease in intensity of certain colors (e.g., red), a facial blood flowpattern for a stronger blood flow will have different contour map thanthe contour map observed in a facial blood flow pattern that correspondsto a weaker blood flow in the same area. The contour map depends on thestructure of the facial blood vessels of the user, and a machinelearning model used to determine extent of blood flow can be trained onsets of images corresponding to different levels of blood flows capturedfrom different users.

Herein, sentences of the form “a facial blood flow pattern recognizablein the images (of an area comprising skin on the user's head)” refer toeffects of blood volume changes due to pulse waves that may be extractedfrom one or more images of the area. These changes may be identifiedand/or utilized by a computer, but need not necessarily be recognizableto the naked eye (e.g., because of their subtlety, the short duration inwhich they occur, or involvement of light outside of the visiblespectrum).

Additionally, sentences of the form “a respiration rate recognizable inthe respiration signal” refer to values that may be extracted from therespiration signal, when algorithms known in the art are utilized tocalculate the respiration rate from the respiration signal.

It is to be noted that stating that a computer performs a calculationbased on a certain value that is recognizable in certain data (e.g., therespiration rate recognizable in the respiration signal) does notnecessarily imply that the computer explicitly extracts the value fromthe data. For example, the computer may perform its calculation withoutexplicitly calculating the respiration rate. Rather, data that reflectsthe respiration rate (the respiration signal) may be provided as inpututilized by a machine learning algorithm. Many machine learningalgorithms (e.g., neural networks) can utilize such an input without theneed to explicitly calculate the value that is “recognizable”.

A facial blood flow pattern, such as one of the examples describedabove, may be calculated, in some embodiments, from the images 821 bythe computer 828. Optionally, the facial blood flow pattern may beutilized to generate one or more feature values that are used in amachine learning-based approach by the computer 828 to calculate theextent of CHF and/or identify an exacerbation of CHF, as describedbelow. In other embodiments, the calculated blood flow pattern may beutilized to calculate additional values used to represent the extent offacial blood flow, which may be evaluated, e.g., by comparing the extentof blood flow to thresholds in order to calculate the extent of CHFand/or identify an exacerbation of CHF.

In one embodiment, a facial blood flow pattern may be converted to avalue representing the proportion of the area in which the intensitiesof pixels reach a threshold. In one example, the intensities beingevaluated may be average intensities (e.g., average pixel intensities inthe images 821). In another example, the intensities being evaluated maybe intensities of systolic peaks (determined by detecting the spread ofa pulse wave in the area captured in the images 821).

In another embodiment, a facial blood flow pattern may be compared withone or more reference facial blood flow patterns that may correspond tospecific blood flow levels and/or specific levels of CHF. Optionally,the reference patterns may be based on previously taken images of theuser, which were taken at times for which the extent of CHF and/or bloodflow was determined manually, e.g., by a physician. Optionally, thesimilarity of the facial blood flow pattern to the reference patternsmay be utilized to generate one or more feature values utilized in amachine learning approach, as described below.

The computer 828 may produce different types of results, in embodimentsdescribed herein, based on input data that includes the images 821 andoptionally other data, such as the respiration signal 823, T_(skin) 825,T_(env) 827, measurements of an IMU 830, and/or additional optional datadescribed herein. Optionally, the results, which may include, forexample, an indication of extent of CHF and/or an indication of theexacerbation, are reported via the user interface 832. Optionally, theresults may be reported to a caregiver (e.g., a physician).

In some embodiments, the computer 828 is configured to calculate theextent of CHF based on: a facial blood flow patterns recognizable in theimages 821, and respiration rate recognizable in the respiration signal823.

In order to calculate the extent of CHF, in one embodiment, the computer828 may compare a value calculated based on the facial blood flowpattern (e.g., a similarity to a reference facial blood flow patter) toa first threshold, and compare the respiration rate to a secondthreshold. Responsive to the value reaching the first threshold, and therespiration rate reaching the second threshold, the computer 828 maydetermine that the user is experiencing CHF to at least a specifiedextent.

In another embodiment, calculating the extent of CHF involves generatingfeature values based on data that includes the facial blood flow patternand the respiration rate. The computer 828 then utilizes a machinelearning model to calculate, based on the feature values, a valueindicative of the extent of CHF the user is experiencing. Additionaldetails regarding the machine learning-based approach are providedfurther below.

Calculating the extent of CHF may involve, in some embodiments,utilization of previous measurements for which a corresponding baselineextent of CHF the user experienced is known. Optionally, the baselineextent of CHF is determined by a physician examining the user.Optionally, one or more feature values utilized to calculate the extentof CHF may be indicative of the baseline value and/or indicative of adifference between the value and a baseline value.

In one embodiment, the computer 828 calculates, based on the images 821,a value indicative of an extent to which skin in the area is blue and/orgray. The computer 828 then utilizes a difference between the value anda baseline value for the extent to which skin in the area is blue and/orgray to calculate the extent of CHF. Optionally, the baseline value wasdetermined while the user experienced a certain baseline extent of CHF.

In another embodiment, the computer 828 calculates, based on the images821, a value indicative of an extent of color changes to skin in thearea due to cardiac pulses. The computer 828 then utilizes a differencebetween the value and a baseline value for the extent of the colorchanges to calculate the extent of CHF. Optionally, the baseline valuefor the extent of the color changes was determined while the userexperienced a certain baseline extent of CHF.

The computer 828 may identify, in some embodiments, an exacerbation ofthe CHF by comparing current measurements of the user, with previousmeasurements of the user. To this end, in one embodiment, the computer828 receives: (i) previous images of the area, which are indicative of aprevious facial blood flow pattern while the user had a certain extentof CHF, and (ii) a previous respiration rate taken while the user hadthe certain extent of CHF. Optionally, the certain extent of CHF isprovided from manual examination of the user, e.g., by a physician whoexamined the user at the time. Additionally or alternatively, thecertain extent of CHF was calculated by the computer 828 based on datathat includes the previous images and the previous respiration signal.

In one embodiment, the computer 828 may identify exacerbation of the CHFbased on: a difference above a first predetermined threshold between thefirst facial blood flow pattern and the previous facial blood flowpattern, and an increase above a second predetermined threshold in therespiration rate compared to the previous respiration rate. Optionally,the first and second thresholds are determined based on previouslyobserved differences in facial blood flow patterns and respirationsignals of the user when the user experienced an exacerbation of CHF.Additionally or alternatively, the first and second thresholds may bedetermined based on differences in facial blood flow patterns andrespiration signals of observed when other users experienced anexacerbation of CHF.

In one embodiment, the computer 828 may identify exacerbation of the CHFutilizing a machine learning based approach, in which feature valuesused to detect an exacerbation of CHF include a feature value isindicative a difference between the first facial blood flow pattern andthe previous facial blood flow patter, and another feature value isindicative of an extent of increase in the respiration rate compared tothe previous respiration rate.

Identifying an exacerbation of CHF may involve, in some embodiments,monitoring the user over a period of multiple days, weeks, and evenmonths or more. Thus, the images 821 and the images and the respirationsignal 823 may be measured over a period spanning multiple days. Inthese embodiments, the computer 828 may identify the exacerbation of theCHF based on a reduction in average facial blood flow recognizable inthe images 821 taken during the period and an increase in the averagerespiration rate recognizable in measurements of the respiration signal823 measured during the period.

One sign of CHF is a difficulty in performing physical activity. Thisleads people who have CHF to be less active. Additionally, when they arephysically active, such as walking a certain distance, the physicalactivity takes a greater than usual toll on their body. This is oftenmanifested as a change in cardio-respiratory activity (such as a smallerincrease in blood flow), which takes longer to return to baseline levelsfrom before conducting the physical activity. Thus, in some embodiments,the computer 828 may utilize measurements taken before and afterphysical activity to calculate an extent of CHF and/or detectexacerbation of CHF.

In order to detect a period in which the user conducted in physicalactivity, and/or what type of physical activity the user conducted, insome embodiments, the computer 828 may utilize measurements of movementsof the user. Optionally, the measurements of movements are obtainedutilizing a movement sensor, such as the IMU 830, which is physicallycoupled to the smartglasses, and configured to measure movements of theuser. In one example, the computer 828 may detect when a period ofphysical activity involves walking for at 20 steps, and comparemeasurements of the user taken before and after the period to determinethe extent of CHF and/or identify exacerbation of CHF, as describedbelow.

In one embodiment, the computer 828 receives a first set of images takenwhile the user was at rest and prior to a period during which the userperformed physical activity. The computer 828 also receives a second setof the images taken within ten minutes after the period. The computer828 calculates the extent of CHF based on differences in facial bloodflow patterns recognizable in the first and second sets of the images.For example, the computer 828 may detect the extent of CHF and/or theexacerbation of CHF based on a determination of whether a difference inparameters of the extent of blood flow which are determined based on thefirst and second sets of images, reaches a threshold. In anotherexample, the computer 828 may generate one or more feature valuesutilized by a machine learning approach described herein, which areindicative of differences between extents of blood flow which aredetermined based on the first and second sets of images.

In one embodiment, the computer 828 calculates a value indicative ofskin color at different times based on the second set of images, andcalculates the extent of CHF based on a length of a duration followingthe period, in which the difference between the skin color and abaseline skin color, calculated based on the first set of images, wasabove a threshold. Optionally, a feature value indicative of the lengthof the duration may be utilized to calculate the extent of CHF.

In another embodiment, the computer 828 calculates a value indicative ofskin color at different times based on the second set of images, andcalculates the extent of CHF based on a rate of return of the user'sskin color to a baseline skin color calculated based on the first set ofimages. Optionally, a feature value indicative of the rate of return maybe utilized to calculate the extent of CHF.

Dynamics of the user's heart rate following physical activity may alsobe used to calculate the extent of CHF. In one embodiment, the computer828 calculates first and second series of heart rate values fromportions of iPPG signals extracted from the first and second sets ofimages, respectively. The computer 828 may calculate the extent of theCHF also based on the extent to which heart rate values in the secondseries were above heart rate values in the first series. For example,the computer 828 may generate one or more feature values indicative ofthese differences, and utilize them in the calculation of the extent ofthe CHF.

In one embodiment, the computer 828 calculates the user's heart rate atdifferent times based on the second sets of images, and calculates theextent of CHF based on a length of a duration following the period, inwhich the difference between the user's heart rate and a baseline heartrate, calculated based on the first set of images, was above athreshold. Optionally, a feature value indicative of the length of theduration may be utilized to calculate the extent of CHF.

Characteristics of respiration before and after physical activity mayalso be indicators of the extent of CHF. In one embodiment, the computer828 receives respiration signal measured while the user was at rest andprior to a period during which the user performed physical activity. Thecomputer 828 also receives a second respiration signal measured withinten minutes after the period. The computer 828 calculates the extent ofCHF based on differences respiration rates recognizable in the first andsecond respiration signals. For example, the computer 828 may detect theextent of CHF, and/or the exacerbation of CHF, based on a determinationof whether a difference in respiration rates determined based on thefirst and second respiration signals, reaches a threshold. In anotherexample, the computer 828 may generate one or more feature valuesutilized by a machine learning approach described herein, which areindicative of differences between the aforementioned respiration rates.

In one embodiment, the computer 828 calculate the user's respirationrate at different times based on the second respiration signal, andcalculate the extent of CHF based on a length of a duration followingthe period, in which the difference between the user's respiration rateand a baseline respiration rate, calculated based on the firstrespiration signal, was above a threshold. Optionally, a feature valueindicative of the length of the duration may be utilized to calculatethe extent of CHF.

The computer 828 may utilize machine learning approaches, in someembodiments, to calculate the extent of CHF and/or identify exacerbationof CHF. In some embodiments, the computer 828 calculates feature valuesbased on data that includes the images 821 (and possibly other data) andutilizes a model to calculate, based on the feature values, a valueindicative of the extent of CHF and/or a value indicative ofexacerbation of CHF (compared to a previous state of the user). Variousexamples of feature values that may be generated based on the images821, the respiration signal 823, and/or measurements of the IMU 830 areprovided above in this disclosure.

Generally, machine learning-based approaches utilized by embodimentsdescribed herein involve training a model on samples, with each sampleincluding: feature values generated based on images taken by theinward-facing camera 820, and optionally respiration signals measuredwith the sensor 822, and/or other data, which were taken during acertain period, and a label indicative of the extent of CHF (during thecertain period). Optionally, a label may be provided manually by theuser and/or by a medical professional who examined the user. Optionally,a label may be extracted based on analysis of electronic health recordsof the user, generated while being monitored at a medical facility.

In some embodiments, the model may be personalized for a user bytraining the model on samples that include: feature values generatedbased on measurements of the user, and corresponding labels indicativeof the extent of CHF experienced by the user at the time. In someembodiments, the model may be generated based on measurements ofmultiple users, in which case, the model may be considered a generalmodel. Optionally, a model generated based on measurements of multipleusers may be personalized for a certain user by being retrained onsamples generated based on measurements of the certain user.

There are various types of feature values that may be generated by thecomputer 828 based on input data, which may be utilized to calculate theextent of CHF and/or identify an exacerbation. Some examples of featurevalues include “raw” or minimally processed values based on the inputdata (i.e., the features are the data itself or applying genericpreprocessing functions to the data). Other examples of feature valuesinclude feature values that are based on higher-level processing, such afeature values determined based on domain-knowledge (e.g., featurevalues describing properties of pulse waveforms) and/or feature valuesthat are based on high-level image-analysis.

The following are some examples of the various types of feature valuesthat may be generated based on the images 821 by the computer 828. Inone embodiment, at least some of the feature values may be deriveddirectly from values of pixels in the images 821. Optionally, at leastsome of the feature values are values of pixels from the images 821.Optionally, one or more of the feature values may be the values of thepixels themselves or some simple function of the pixels, such as theaverage of pixels at certain regions in each of the images. Optionally,one or more of the feature values may be various low-level featuresderived from images, such as features generated using Gabor filters,local binary patterns (LBP) and their derivatives, algorithms such asSIFT and/or SURF (and their derivatives), image keypoints, histograms oforiented gradients (HOG) descriptors, and products of statisticalprocedures such independent component analysis (ICA), principalcomponent analysis (PCA), or linear discriminant analysis (LDA).Optionally, one or more of the feature values may derived from multipleimages taken at different times, such as volume local binary patterns(VLBP), cuboids, and/or optical strain-based features. In one example,one or more of the feature values may represent a difference betweenvalues of pixels at one time t at a certain location on the face andvalues of pixels at a different location at some other time t+x (whichcan help detect different arrival times of a pulse wave).

In some embodiments, at least some feature values utilized by thecomputer 828 to calculate the extent of CHF and/or identify exacerbationof CHF describe properties of the cardiac waveform in an iPPG signalderived from the images 821. To this end, the computer 828 may employvarious approaches known in the art to identify landmarks in a cardiacwaveform (e.g., systolic peaks, diastolic peaks), and/or extract varioustypes of known values that may be derived from the cardiac waveform, asdescribed in the following examples.

In one embodiment, at least some of the feature values generated basedon the iPPG signal may be indicative of waveform properties thatinclude: systolic-upstroke time, diastolic time, and the time delaybetween the systolic and diastolic peaks, as described in Samria, Rohan,et al. “Noninvasive cuffless estimation of blood pressure usingPhotoplethysmography without electrocardiograph measurement.” 2014 IEEEREGION 10 SYMPOSIUM. IEEE, 2014.

In another embodiment, at least some of the feature values generatedbased on the iPPG signal may be derived from another analysis approachto PPG waveforms, as described in US Patent Application US20180206733,entitled “Device, method and system for monitoring and management ofchanges in hemodynamic parameters”, which was published on 26 Jul. 2018.This approach assumes the cardiac waveform has the following structure:a minimum/starting point (A), which increases to a systolic peak (B),which decreases to a dicrotic notch (C), which increases to a dicroticwave (D), which decreases to the starting point of the next pulse wave(E). Various features that may be calculated by the computer 828, whichare suggested in the aforementioned publication, include: value of A,value of B, value of C, value of D, value of E, systol area that is thearea under ABCE, diastol area that is the area under CDE, and the ratiobetween BC and DC.

In still another embodiment, the computer 828 may utilize the variousapproaches described in Elgendi, M. (2012), “On the analysis offingertip photoplethysmogram signals”, Current cardiology reviews, 8(1),14-25, in order to generate at least some of the feature values bases onthe iPPG signal. This reference surveys several preprocessing approachesfor PPG signals as well as a variety of feature values that may beutilized. Some of the techniques described therein, which may beutilized by the computer 828, include calculating feature values basedon first and second derivatives of PPG signals.

In some embodiments, at least some of the feature values may representcalibration values of a user, which are values of certain parameterssuch as waveform properties described above when the user had a knownextent of CHF (e.g., as determined based on an evaluation by aphysician). Optionally, the computer 828 generates one or more valuesthat are indicative of: (i) a value of the extent of CHF experienced bythe user during a certain previous period, and (ii) a value of aproperty of the pulse waveform (e.g., systolic-upstroke time ordiastolic time) during the certain previous period.

One sign of CHF is an increase heart rate. In some embodiments, thecomputer 828 may utilize one or more feature values indicative of theuser's heart rate. Optionally, these feature values may be derived fromthe images 821, e.g., by performing calculations on iPPG signalsextracted from the images 821. In one example, U.S. Pat. No. 8,768,438,titled “Determining cardiac arrhythmia from a video of a subject beingmonitored for cardiac function”, describes how to obtain a PPG signalfrom video of a user's face. In this example, a time series signal isgenerated from video images of a subject's exposed skin, and a referencesignal is used to perform a constrained source separation (which is avariant of ICA) on the time series signals to obtain the PPG signal.Peak-to-peak pulse points are detected in the PPG signal, which may beanalyzed to determine parameters such as heart rate, heart ratevariability, and/or to obtain peak-to-peak pulse dynamics that can beindicative of conditions such as cardiac arrhythmia.

In some embodiments, the computer 828 generates at least some featurevalues utilized to calculate the extent of CHF, and/or identifyexacerbation of CHF, based on the respiration signal 823. In someembodiments, these feature values may be “raw” or minimally processedvalues. In one example, in which the respiration signal 823 includesthermal measurements, the feature values may be values of themeasurements themselves and/or various functions and/or statistics ofthe thermal measurements such as minimum/maximum measurement valuesand/or average values during certain windows of time. In anotherexample, in which the respiration signal 823 includes audio, the featurevalues may include various acoustic features derived from the audio. Forexample, the audio may be represented as a time series of vectors ofacoustic features, where each vector corresponds to a short window ofthe audio. For example, windows may be between κ ms and 200 ms long. Thesignal in a window may be processed in various ways to obtain acousticfeatures. In one example, fast Fourier transform (FFT) is performed onthe audio in each window. From the FFT data for each window, variousfeatures may be extracted. For example, some acoustic features may bedetermined by binning according to filterbank energy coefficients, usinga Mel-frequency cepstral component (MFCC) transform, using a perceptuallinear prediction (PLP) transform, or using other techniques.

In some embodiments, the computer 828 generates at least some featurevalues utilized to calculate the extent of CHF and/or identifyexacerbation of CHF based on values of respiration parameters calculatedbased on the respiration signal 823 (e.g., the respiration rate and/orother respiration parameters). Some examples of respiration parametersthat may be calculated based on thermal measurements include: breathingrate, respiration volume, whether the user is breathing mainly throughthe mouth or through the nose, exhale (inhale) duration, post-exhale(post-inhale) breathing pause, a dominant nostril, a shape of the exhalestream, smoothness of the exhale stream, and/or temperature of theexhale stream. Additional details regarding calculation of theseparameters is provided in U.S. Pat. No. 10,130,308, titled “Calculatingrespiratory parameters from thermal measurements”.

In one non-limiting example, feature values generated by the computer828 include pixel values from the images 821 and values obtained bybinning according to filterbank energy coefficients, using MFCCtransform on results of FFT of the respiration signal 823, whichincludes audio.

In another non-limiting example, feature values generated by thecomputer 828 include timings and intensities corresponding to fiducialpoints identified in iPPG signals extracted from the images 821, andvalues of respiration parameters calculated based on the respirationsignal 823.

In some embodiments, one or more of the feature values utilized by thecomputer 828 to calculate the extent of CHF and/or identify exacerbationof CHF may be generated based on additional inputs from sources otherthan the inward-facing camera 820 or the sensor 822.

Stress is a factor that can influence the diameter of the arteries, andthus influence the blood flow. In one embodiment, the computer 828 isfurther configured to: receive a value indicative of a stress level ofthe user, and generate at least one of the feature values based on thereceived value. Optionally, the value indicative of the stress level isobtained using a thermal camera. In one example, the system may includean inward-facing head-mounted thermal camera configured to takemeasurements of a periorbital region of the user, where the measurementsof a periorbital region of the user are indicative of the stress levelof the user. In another example, the system includes an inward-facinghead-mounted thermal camera configured to take measurements of a regionon the forehead of the user, where the measurements of the region on theforehead of the user are indicative of the stress level of the user. Instill another example, the system includes an inward-facing head-mountedthermal camera configured to take measurements of a region on the noseof the user, where the measurements of the region on the nose of theuser are indicative of the stress level of the user.

Hydration is a factor that affects blood viscosity, which can affect thespeed at which blood flows in the body, and consequently may affectblood flow patterns recognizable in the images 821. In one embodiment,the computer 828 is further configured to: receive a value indicative ofa hydration level of the user, and generate at least one of the featurevalues based on the received value. Optionally, the system includes anadditional camera configured to detect intensity of radiation that isreflected from a region of exposed skin of the user, where the radiationis in spectral wavelengths chosen to be preferentially absorbed bytissue water. In one example, said wavelengths are chosen from threeprimary bands of wavelengths of approximately 1100-1350 nm,approximately 1500-1800 nm, and approximately 2000-2300 nm. Optionally,measurements of the additional camera are utilized by the computer 828as values indicative of the hydration level of the user.

Momentary physical activity can affect the blood flow of the user (e.g.,due to the increase in the heart rate that it involves). In order toaccount for this factor, in some embodiments, the computer 828 maygenerate one or more feature values representing the extent of theuser's movement from measurements of the IMU 830. In addition, theextent of movement of the user may be indicative of the extent of CHF,since users with advanced CHF tend to be less physically active becausephysical activity becomes more difficult due to their condition. Thus,the extent of physical activity (e.g., how many steps a user walked in aday), which may also be derived from measurements of the IMU 830, may beutilized to generate one or more feature values, in some embodiments.

The user's skin temperature may affect blood viscosity, thus it mayinfluence facial blood flow patterns that are recognizable in imagestaken by the inward-facing camera 820. Some embodiments may include theskin temperature sensor 824, which may be a head-mounted sensorphysically coupled to the smartglasses. The skin temperature sensor 824measures temperature of a region comprising skin on the user's head(T_(skin) 825). In one embodiments, the computer 828 is configured toutilize T_(skin) 825 to compensate for effects of skin temperature onthe facial blood flow pattern. For example, the computer 828 maygenerate one or more feature values based on T_(skin) 825, such asfeature values indicating average skin temperature or a difference frombaseline skin temperature, and utilize these one or more feature valuesto calculate the extent of CHF and/or identify exacerbation of CHF.

The temperature in the environment may also be a factor that isconsidered in some embodiments. The temperature in the environment canboth impact the user's skin temperature and cause a physiologic responseinvolved in regulating the user's body temperature on the facial bloodflow pattern. Some embodiments may include the environment temperaturesensor 826, which may optionally, be head-mounted (e.g., physicallycoupled to the smartglasses). The environment temperature sensor 826measures an environmental temperature (T_(env) 827). In one embodiment,the computer 828 is configured to utilize T_(env) 827 to compensate foreffects of physiologic changes related to regulating the user's bodytemperature on the facial blood flow pattern. For example, the computer828 may generate one or more feature values based on T_(env) 827, suchas feature values indicating average environment temperature, maximalenvironment temperature, or a difference from baseline environmenttemperature, and utilize these one or more feature values to calculatethe extent of CHF and/or identify exacerbation of CHF.

Training the model utilized to calculate the extent of CHF and/oridentify an exacerbation of CHF may involve utilization of varioustraining algorithms known in the art (e.g., algorithms for trainingneural networks, and/or other approaches described herein). After themodel is trained, feature values may be generated for certainmeasurements of the user (e.g., the images 821, respiration signal 823,etc.), for which the value of the corresponding label (e.g., the extentof CHF and/or indicator of whether there is an exacerbation) is unknown.The computer 828 can utilize the model to calculate the extent of CHF,and/or whether there is an exacerbation of CHF, based on these featurevalues.

In some embodiments, the model may be generated based on data thatincludes measurements of the user (i.e., data that includes images takenby the inward-facing camera 820, the sensor 822, and/or other sensorsmentioned herein). Additionally or alternatively, in some embodiments,the model may be generated based on data that includes measurements ofone or more other users (such as users of different ages, weights,sexes, body masses, and health states). In order to achieve a robustmodel, which may be useful for calculating the extent of CHF in variousconditions, in some embodiments, the samples used to train the model mayinclude samples based on measurements taken in different conditions.Optionally, the samples are generated based on measurements taken ondifferent days, while in different locations, and/or while differentenvironmental conditions persisted. In a first example, the model istrained on samples generated from a first set of measurements takenwhile the user was indoors and not in direct sunlight, and is alsotrained on other samples generated from a second set of measurementstaken while the user was outdoors, in direct sunlight. In a secondexample, the model is trained on samples generated from a first set ofmeasurements taken during daytime, and is also trained on other samplesgenerated from a second set of measurements taken during nighttime. In athird example, the model is trained on samples generated from a firstset of measurements taken while the user was moving, and is also trainedon other samples generated from a second set of measurements taken whilethe user was sitting.

Utilizing the model to calculate the extent of CHF and/or identifyexacerbation of CHF may involve the computer 828 performing variousoperations, depending on the type of model. The following are someexamples of various possibilities for the model and the type ofcalculations that may be accordingly performed by the computer 828, insome embodiments, in order to calculate extent of CHF and/or identifyexacerbation of CHF: (a) the model comprises parameters of a decisiontree. Optionally, the computer 828 simulates a traversal along a path inthe decision tree, determining which branches to take based on thefeature values. A value indicative of the extent of CHF may be obtainedat the leaf node and/or based on calculations involving values on nodesand/or edges along the path; (b) the model comprises parameters of aregression model (e.g., regression coefficients in a linear regressionmodel or a logistic regression model). Optionally, the computer 828multiplies the feature values (which may be considered a regressor) withthe parameters of the regression model in order to obtain the valueindicative of the extent of CHF; and/or (c) the model comprisesparameters of a neural network. For example, the parameters may includevalues defining at least the following: (i) an interconnection patternbetween different layers of neurons, (ii) weights of theinterconnections, and (iii) activation functions that convert eachneuron's weighted input to its output activation. Optionally, thecomputer 828 provides the feature values as inputs to the neuralnetwork, computes the values of the various activation functions andpropagates values between layers, and obtains an output from thenetwork, which is the value indicative of the extent of the CHF and/oran indicator of whether there was an exacerbation of the CHF.

In some embodiments, a machine learning approach that may be applied tocalculating extent of CHF and/or identifying exacerbation of CHF basedon images may be characterized as “deep learning”. In one embodiment,the model may include parameters describing multiple hidden layers of aneural network. Optionally, the model may include a convolution neuralnetwork (CNN). In one example, the CNN may be utilized to identifycertain patterns in the video images, such as the facial blood flowpatterns involving blood volume effects and ballistocardiographiceffects of the cardiac pulse. Due to the fact that calculations areperformed on sequences images display a certain pattern of change overtime (i.e., across multiple frames), these calculations may involveretaining state information that is based on previous images in thesequence. Optionally, the model may include parameters that describe anarchitecture that supports such a capability. In one example, the modelmay include parameters of a recurrent neural network (RNN), which is aconnectionist model that captures the dynamics of sequences of samplesvia cycles in the network's nodes. This enables RNNs to retain a statethat can represent information from an arbitrarily long context window.In one example, the RNN may be implemented using a long short-termmemory (LSTM) architecture. In another example, the RNN may beimplemented using a bidirectional recurrent neural network architecture(BRNN).

The following is description of additional aspects of embodiments ofsystems configured to detect an abnormal medical event, as well asadditional embodiments for various systems that may detect physiologicalresponses based on sensor measurements and/or other sources of data.

In some embodiments, a system configured to detect an abnormal medicalevent includes a computer and several head-mounted devices that are usedto measure photoplethysmographic signals (PPG signals) indicative ofblood flow at various regions on a user's head. Optionally, the systemmay include additional components, such as additional sensors that maybe used to measure the user and/or the environment. Additionally oralternatively, the system may include a frame configured to be worn onthe user's head (e.g., a frame of eyeglasses or smartglasses) and towhich at least some of the head-mounted devices and sensors arephysically coupled. Having sensors, such as the devices that are used tomeasure photoplethysmographic signals, physically coupled to the framemay convey certain advantages, such as having the sensors remain at thesame positions with respect to the head, even when the user's head makesangular movements. Some examples of abnormal medical events that may bedetected by embodiments described herein include an ischemic stroke, amigraine, a headache, cellulitis (soft tissue infection), dermatitis(skin infection), and an ear infection.

In one embodiment, the system includes at least one right-sidehead-mounted device, configured to measure at least two signalsindicative of photoplethysmographic signals (PPG_(SR1) and PPG_(SR2),respectively) at first and second regions of interest (ROI_(R1) andROI_(R2), respectively) on the right side of a user's head. Optionally,ROI_(R1) and ROI_(R2) are located at least 2 cm apart (where cm denotescentimeters). Optionally, each device, from among the at least oneright-side head-mounted device(s), is located to the right of thevertical symmetry axis that divides the user's face.

Additionally, the system includes at least one left-side head-mounteddevice configured to measure at least two signals indicative ofphotoplethysmographic signals (PPG_(SL1) and PPG_(SL2), respectively) atfirst and second regions of interest (ROI_(L1) and ROI_(L2),respectively) on the left side of the user's head. Optionally, eachdevice, from among the at least one right-side head-mounted device(s),is located to the left of the vertical symmetry axis that divides theuser's face.

In some embodiments, ROI_(R1) and ROI_(L1) may be symmetrical regions onthe right and left sides of the head, respectively (with respect to asymmetry axis that splits the face to right and left sides).Additionally or alternatively, ROI_(R2) and ROI_(L2) may be symmetricalregions on the right and left sides of the head, respectively. In otherembodiments, ROI_(R1) and ROI_(L1) may not be symmetrical regions on theright and left side of the head, and/or ROI_(R2) and ROI_(L2) may not besymmetrical regions on the right and left side of the head. Optionally,two regions are considered to be in symmetrical locations if one regionis within 1 cm of the symmetrical region on the head.

Various types of devices may be utilized in order to obtain PPG_(SR1),PPG_(SR2), PPG_(SL1), and/or PPG_(SL2). In one embodiment, the at leastone right-side head-mounted device includes first and second contactphotoplethysmographic devices (PPG₁, PPG₂, respectively). Additionallyor alternatively, the at least one left-side head-mounted device mayinclude third and fourth contact photoplethysmographic devices (PPG₃,PPG₄, respectively). Herein, a “contact photoplethysmographic device” isa photoplethysmographic device that comes in contact with the user'sskin, and typically occludes the area being measured. An example of acontact photoplethysmographic device is the well-known pulse oximeter.

In one example, PPG₁, PPG₂, PPG₃, and PPG₄ are physically coupled to aneyeglasses frame, PPG₁ and PPG₃ are in contact with the nose, and PPG₂and PPG₄ are in contact with regions in the vicinities of the ears(e.g., within a distance of less than 5 cm from the center of theconcha), respectively.

FIG. 3a illustrates smartglasses that include contactphotoplethysmographic devices that may be used to obtain PPG signals, asdescribed above. The contact PPG devices are coupled to a frame 674. Thecontact PPG devices may be coupled at various locations on the frame 674and thus may come in contact with various regions on the user's head.For example, contact PPG device 671 a is located on the right templetip, which brings it to contact with a region behind the user's ear(when the user wears the smartglasses). Contact PPG device 671 b islocated on the right temple of the frame 674, which puts it in contactwith a region on the user's right temple (when wearing thesmartglasses). It is to be noted that in some embodiments, in order tobring the contact PPG device close such that it touches the skin,various apparatuses may be utilized, such as spacers (e.g., made fromrubber or plastic), and/or adjustable inserts that can help bridge apossible gap between a frame's temple and the user's face. Such anapparatus is spacer 672 which brings contact PPG device 671 b in contactwith the user's temple when the user wears the smartglasses.

Another possible location for a contact PPG device is the nose bridge,as contact PPG device 671 c is illustrated in the figure. It is to benoted the contact PPG device 671 c may be embedded in the nose bridge(or one of its components), and/or physically coupled to a part of thenose bridge. The figure also illustrates computer 673, which may beutilized in some embodiments, to perform processing of PPG signalsand/or detection of the abnormal medical event, as described furtherbelow.

It is to be noted that FIG. 3a illustrates but a few of the possiblelocations for contact PPG devices on a frame. Any pair of contact PPGdevices 671 a, 671 b, and 671 c may be the aforementioned PPG₁ and PPG₂.Additionally, some embodiments may include additional contact PPGdevices on each side of the frames. Furthermore, it is to be noted thatwhile contact PPG devices on the left side were not illustrated in thefigure, additional PPG devices may be located in similar locations tothe ones of PPG devices 671 a to 671 c are located, but on the left sideof the frame.

In another embodiment, one or more video cameras may be utilized toobtain PPG_(SR1), PPG_(SR2), PPG_(SL1), and/or PPG_(SL2) utilizingimaging photoplethysmography. Using video cameras can be advantageous insome scenarios, such as stroke, where it is unknown in advanced wherethe physiological response associated with the abnormal medical eventwill appear on the user's head. Thus, in some embodiments involvingthese scenarios, using video cameras may provide a great advantage overcontact PPG devices, because the video cameras cover larger areas, whichincrease the chance to capture on time the physiological responseassociated with the abnormal medical event.

In one embodiment, the at least one right-side head-mounted deviceincludes a first inward-facing camera located more than 0.5 cm away fromROI_(R1) and ROI_(R2), and PPG_(SR1) and PPG_(SR2) are recognizable fromcolor changes in regions in images taken by the first camera.Additionally or alternatively, the at least one left-side head-mounteddevice may include a second inward-facing camera located more than 0.5cm away from ROI_(L1) and ROI_(L2), and PPG_(SL1) and PPG_(SL2) arerecognizable from color changes in regions in images taken by the secondcamera. In one embodiment, the system includes both at least one contactPPG device and at least one video camera.

In one embodiment, each of the first and second inward-facing camerasutilizes a sensor having more than 30 pixels, and each of ROI_(R1) andROI_(L1) covers a skin area greater than 6 cm{circumflex over ( )}2,which is illuminated by ambient light. In another embodiment, each ofthe first and second inward-facing cameras utilizes a sensor having morethan 20 pixels, and each of ROI_(R1) and ROI_(L1) covers a skin areagreater than 2 cm{circumflex over ( )}2, which is illuminated by ambientlight. In still another embodiment, each of the first and secondinward-facing cameras utilizes a sensor comprising at least 3×3 pixelsconfigured to detect electromagnetic radiation having wavelengths in atleast a portion of the range of 200 nm to 1200 nm. Optionally, thesystem includes first and second active light sources configured toilluminate portions of the right side of the face (which includeROI_(R1) and ROI_(R2)) and portions of the left side of the face (whichinclude ROI_(L1) and ROI_(L2)), respectively. In one example, the firstand second active light sources are head-mounted light sourcesconfigured to illuminate their respective portions of the face withelectromagnetic radiation having wavelengths in at least a portion ofthe range of 750 nm to 1200 nm.

In one embodiment, due to the angle between the optical axis of acertain inward-facing camera (from among the first and secondinward-facing cameras) and its ROI, the Scheimpflug principle may beemployed in order to capture sharper images with the certaininward-facing camera. For example, when the user wears a frame to whichthe certain inward-facing camera is coupled, the certain inward-facingcamera may have a certain tilt greater than 2° between its sensor andlens planes, in order to capture the sharper images.

FIG. 3b illustrates smartglasses that include first and secondinward-facing cameras, such as the ones described above. The figureillustrates a frame 677 to which a first inward-facing camera 675 a iscoupled above the lens that is in front of the right eye, and a secondinward-facing camera 675 b that is coupled to the frame 677 above thelens that is in front of the left eye. The figure also illustrates acomputer 676 that is coupled to the frame 677, and may be utilized toprocess images obtained by the first inward-facing camera 675 a and/orthe second inward-facing camera 675 b, and/or perform the detection ofthe abnormal medical event based on PPG signals recognizable in imagescaptured by those cameras.

Various regions on the face may be measured in embodiments that utilizeimaging photoplethysmography. In one example, ROI_(R1) and ROI_(L1) arelocated on the user's right and left cheeks, respectively. In anotherexample, ROI_(R2) and ROI_(L2) are located on the right and left sidesof the user's nose, respectively. In yet another example, ROI_(R1) andROI_(L1) are located on the right and left sides of the user's forehead,respectively. In still another example, ROI_(R2) and ROI_(L2) arelocated on the user's right and left temples, respectively.

A configuration consistent with some of the examples described above isillustrated in FIG. 4. In this figure, four inward-facing cameras arecoupled to a frame 540 worn on a user's head: (i) Inward-facing camera544 a is coupled to the top-left side of the frame and captures imagesof ROI 545 a, which is on the left side of the user's forehead; (ii)Inward-facing camera 544 b is coupled to the top-right side of the frameand captures images of ROI 545 b, which is on the right side of theuser's forehead; (iii) Inward-facing camera 546 a is coupled to thebottom-left side of the frame and captures images of ROI 547 a, which ison the user's left cheek; And (iv) Inward-facing camera 546 b is coupledto the bottom-right side of the frame and captures images of ROI 547 b,which is on the user's right cheek.

Herein, sentences of the form “a PPG signal is recognizable from colorchanges in a region in images” refer to effects of blood volume changesdue to pulse waves that may be extracted from a series of images of theregion. These changes may be identified and/or utilized by a computer(e.g., in order to generate a signal indicative of the blood volume atthe region), but need not necessarily be recognizable to the naked eye(e.g., because of their subtlety, the short duration in which theyoccur, or involvement of light outside of the visible spectrum). Forexample, blood flow may cause facial skin color changes (FSCC) thatcorresponds to different concentrations of oxidized hemoglobin due tovarying volume of blood at a certain region due to different stages of acardiac pulse, and/or the different magnitudes of cardiac output.Similar blood flow dependent effects may be viewed with other types ofsignals (e.g., slight changes in cutaneous temperatures due to the flowof blood).

In some embodiments, the system configured to detect the abnormalmedical event may further include first and second outward-facinghead-mounted cameras for taking images of the environment to the rightand to the left of the user's head, respectively. Images taken by thesecameras, which are indicative of illumination towards the face, may beutilized to improve the accuracy of detecting the abnormal medicalevent.

In one example, the first and second outward-facing head-mounted camerasmay be thermal cameras that take thermal measurements of theenvironment. Heat from the environment may affect the surface bloodflow, and thus reduce the accuracy of detecting the abnormal medicalevent. By taking the thermal measurements of the environment intoaccount, the computer is able to detect, and maybe even compensate, fortemperature interferences from the environment. Examples ofoutward-facing head-mounted thermal cameras include thermopile-basedand/or microbolometer-based cameras having one or more pixels.

In another example, the first and second outward-facing head-mountedcameras may be visible-light cameras (such as CMOS cameras), and/orlight intensity sensors (such as photodiodes, photoresistors, and/orphototransistor). Illumination from the environment may affect thesurface blood flow (especially when heating the skin), and/or interferewith the photoplethysmographic signals to be measured, and thus reducethe accuracy of detecting the abnormal medical event. By taking theillumination from the environment into account, the computer is able todetect, and maybe even compensate, for the interferences from theenvironment.

In one embodiment, the at least one right-side head-mounted deviceand/or the at least one left-side head-mounted device are coupled to aclip-on, and the clip-on comprises a body configured to be attached anddetached, multiple times, from a frame configured to be worn on theuser's head. Various embodiments described herein include a computerconfigured to detect the abnormal medical event based on an asymmetricalchange to blood flow recognizable in at least PPG_(SR1), PPG_(SR2),PPG_(SL1), and PPG_(SL2). Optionally, the asymmetrical change to theblood flow corresponds to a deviation of PPG_(SR1), PPG_(SR2),PPG_(SL1), and PPG_(SL2) compared to a baseline based on previousmeasurements of PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2) of theuser, taken before the abnormal medical event (e.g., minutes, hours, andeven days before the abnormal medical event). In one example, “abaseline based on the previous measurements” is one or more values thatare calculated based on the previous measurements (e.g., one or morevalues representing a normal, baseline blood flow of the user). Inanother example, “a baseline based on the previous measurements” may besome, or even all, the previous measurements themselves, which may beprovided as an input used in calculations involved in the detection ofthe abnormal medical event (without necessarily calculating an explicitvalue that is considered a “baseline” value of the user's blood flow).

Examples of computers that may be utilized to perform calculationsinvolved in the detection of the abnormal medical event are computersmodeled according to computer 400 or computer 410 illustrated in FIG.23a and FIG. 23b , respectively. Additional examples are the computers673 and 676 illustrated in FIG. 3a and FIG. 3b , respectively. It is tobe noted that the use of the singular term “computer” is intended toimply one or more computers, which jointly perform the functionsattributed to “the computer” herein. In particular, in some embodiments,some functions attributed to the computer, such as preprocessingPPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2), may be performed by aprocessor on a wearable device (e.g., smartglasses) and/or a computingdevice of the user (e.g., smartphone), while other functions, such asthe analysis of sensor data and determining whether the user isexperiencing the abnormal medical event, may be performed on a remoteprocessor, such as a cloud-based server. In other embodiments,essentially all functions attributed to the computer herein may beperformed by a processor on a wearable device (e.g., smartglasses towhich the head-mounted devices are coupled) and/or some other devicecarried by the user, such as a smartwatch or smartphone.

Herein, detecting the abnormal medical event may mean detecting that theuser is suffering from the abnormal medical event, and/or that there isan onset of the abnormal medical event. Additionally, an “abnormal”medical event may be a medical event that the user has yet toexperience, or does not experience most of the time.

In some embodiments, detecting the abnormal medical event may involvecalculating one or more of the following values: an indication ofwhether or not the user is experiencing the abnormal medical event, avalue indicative of an extent to which the user is experiencing theabnormal medical event, a duration since the onset of the abnormalmedical event, and a duration until an onset of the abnormal medicalevent.

When the blood flow on both sides of the head and/or body are monitored,asymmetric changes may be recognized. These changes are typicallydifferent from symmetric changes that can be caused by factors such asphysical activity (which typically affects the blood flow on both sidesin the same way). An asymmetric change to the blood flow can mean thatone side has been affected by an event, such as a stroke, which does notinfluence the other side. In one example, the asymmetric change to bloodflow involves a change in blood flow velocity on left side of the headthat is at least 10% greater or 10% lower than a change in blood flowvelocity on one right side of the head. In another example, theasymmetric change to blood flow involves a change in the volume of bloodthe flows during a certain period in the left side of the head that isat least 10% greater or 10% lower than the volume of blood that flowsduring the certain period in the right side of the head. In yet anotherexample, the asymmetric change to blood flow involves a change in thedirection of the blood flow on one side of the head (e.g., as a resultof a stroke), which is not necessarily observed at the symmetriclocation on the other side of the head.

Referring to an asymmetrical change to blood flow as being “recognizablein PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2)” means that valuesextracted from PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2) provide anindication that an asymmetric change to the blood flow has occurred.That is, a difference that has emerged in the PPG_(SR1), PPG_(SR2),PPG_(SL1), and PPG_(SL2) may reflect a change in blood flow velocity onone side of the head, a change in blood flow volume, and/or a change inblood flow direction, as described in the examples above. It is to benoted, that the change in blood flow does not need to be directlyquantified from the values PPG_(SR1), PPG_(SR2), PPG_(SL1), andPPG_(SL2) in order for it to be “recognizable in PPG_(SR1), PPG_(SR2),PPG_(SL1), and PPG_(SL2)”. Rather, in some embodiments, feature valuesgenerated based on PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2) may beused by a machine learning-based predictor to detect a phenomenon, suchas the abnormal medical event, which is associated with the asymmetricalchange in blood flow.

Having multiple PPG signals, measured at different sides of the head,can assist in detecting an asymmetric change to blood flow in differentways. In one example, an asymmetric change in blood flow may becharacterized in an increase in volume at a certain region and/or sideof the head, compared to other regions and/or the other side of thehead. In this example, the amplitude of the PPG signal at the certainregion may show a greater increase compared to increases observed withPPG signals at other regions and/or on the other side of the head. Inanother example, an asymmetric change in blood flow may be characterizedby a decrease in blood velocity at a certain region and/or side of thehead, compared to other regions and/or the other side of the head. Inthis example, a pulse arrival time (PAT) at the certain region mayexhibit a larger delay compared to the delay of PATs at other regionsand/or at the other side of the head. In still another example, anasymmetric change in blood flow may be characterized by a change in adirection of blood flow at a certain region, compared to other regionsand/or the symmetric region on the other side of the head. In thisexample, an order at which pulse waves arrive at different regions (asevident by PPG signals at the different regions) may be indicative ofthe direction of blood flow. Thus, a change in the arrival order ofpulse waves on one side of the head, which does not occur at regions onthe other side of the head, may indicate an asymmetrical change of adirection of blood flow.

In some embodiments, the computer detects the abnormal medical event byutilizing previously taken PPG signals of the user (i.e., previouslytaken PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2)), from a periodthat precedes the current abnormal medical event being detected at thattime. This enables an asymmetrical change to be observed, since itprovides a baseline according to which it is possible to compare currentPPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2), such that it may bedetermined that a change to blood flow on one side of the head is notthe same as a change on the other side of the head. In some embodiments,previously taken PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2) areutilized to calculate a baseline blood flow, such as values representingthe extent of blood flow at the different sides of the head, and/or atdifferent regions (e.g., ROI_(R1), ROI_(R2), ROI_(L1) and ROI_(L2)).Optionally, calculating the baseline blood flow may be done based onpreviously taken PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2) thatwere measured at least an hour before the abnormal medical event isdetected. Optionally, the previously taken PPG_(SR1), PPG_(SR2),PPG_(SL1), and PPG_(SL2) include PPG signals measured at least a daybefore the abnormal medical event is detected. Optionally, thepreviously taken PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2) includePPG signals measured more than a week before the abnormal medical eventis detected.

A baseline for the blood flow may be calculated in various ways. In afirst example, the baseline is a function of the average measurements ofthe user (which include previously taken PPG_(SR1), PPG_(SR2),PPG_(SL1), and PPG_(SL2)), which were taken before the occurrence of theabnormal medical event. In a second example, the baseline may be afunction of the situation the user is in, such that previousmeasurements taken during similar situations are weighted higher thanprevious measurements taken during less similar situations. A PPG signalmay show different characteristics in different situations because ofthe different mental and/or physiological states of the user in thedifferent situations. As a result, a situation-dependent baseline canimprove the accuracy of detecting the abnormal medical event. In a thirdexample, the baseline may be a function of an intake of some substance,such that previous measurements taken after consuming similar substancesare weighted higher than previous measurements taken after not consumingthe similar substances and/or after consuming less similar substances. APPG signal may show different characteristics after the user consumesdifferent substances because of the different mental and/orphysiological states the user may be in after consuming the substances,especially when the substances include things such as medications,drugs, alcohol, and/or certain types of food. As a result, asubstance-dependent baseline can improve the accuracy of detecting theabnormal medical event.

There are various types of abnormal medical events that may be detectedbased on PPG signals that reflect an asymmetrical change to blood flow,which is recognizable in PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2)of the user.

In some embodiments, the abnormal medical event may involve the userhaving a cerebrovascular accident, which is also known as having astroke. An occurrence of a stroke often has the following effect on aperson. A blood clot (in the case of ischemic stroke) or the rupturedartery (in the case of a hemorrhagic stroke) changes the blood flow tocertain regions of the brain. One or more of several mechanisms may bethe cause of changes to blood flow that are observed following an onsetof a stroke. Blood flow may change due to a stroke because of flaccidmuscles (on one side of the face) that use less oxygen and demand lessblood. In such an event, local regulation mechanisms may generatesignals to the smooth muscles that decrease the diameter of the arteries(which can reduce blood flow). Additionally or alternatively, blood flowmay change due to a stroke because of nerve control changes that occurdue to reduced blood flow to the brain (a neurogenic mechanism); thesame nerves that control the muscles can also be involved in the controlof the constriction/dilation of blood vessels. Another possible cause ofchanges to blood flow involves obstruction-related passive changes.Blood that flows through the major vessels (in the base of the brain itis either the carotid (front) or vertebral (back) arteries, must flowout through one of the branches. When one pathway is blocked orrestricted (due to the stroke), more blood has to go through collateralpathways (which may change the blood flow). Thus, changes to the bloodflow in the face (and other areas of the head), especially if they areasymmetric, can be early indicators of a stroke. An event of a strokemay be detected in various ways, some which are described in thefollowing examples.

In one embodiment, the abnormal medical event is an ischemic stroke, andthe deviation (of PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2)compared to a baseline based on the previous measurements of PPG_(SR1),PPG_(SR2), PPG_(SL1), and PPG_(SL2) of the user) involves an increase inasymmetry between blood flow on the left side of the head and blood flowon the right side of the head, with respect to a baseline asymmetrylevel between blood flow on the left side of the head and blood flow onthe right side of the head (as determined based on the previousmeasurements). In some embodiments, the term “ischemic stroke” may alsoinclude Transient Ischemic Attack (TIA), known as “mini stroke”.

In another embodiment, the abnormal medical event is an ischemic stroke,and the deviation (of PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2)compared to a baseline based on the previous measurements of PPG_(SR1),PPG_(SR2), PPG_(SL1), and PPG_(SL2) of the user) involves a monotonicincrease in asymmetry between blood flow at ROI_(R1) and ROI_(R2), withrespect to a baseline asymmetry of the user (between blood flow atROI_(R1) and ROI_(R2)) during a period longer than 10 minutes.Optionally, ROI_(R1) is located in proximity of the mastoid processbehind the right ear, and ROI_(R2) is located before of the right ear.

In yet another embodiment, the abnormal medical event is an ischemicstroke, and the deviation involves an increase in variation betweenblood flow at ROI_(R1) and ROI_(R2), with respect to a baselinevariation of the user between blood flow at ROI_(R1) and ROI_(R2).Optionally, the computer suggests the user to take images of at leastone of the retinas, responsive to detecting the increase in variation.The computer may then compare the images of the retinas with previouslytaken images of the user's retinas, and utilize such a comparison toimprove the accuracy of detecting whether the user has suffered theischemic stroke. Optionally, the comparison of the images of the retinasmay take into account parameters such as the diameter of retinalarteries, swelling of the boundaries of the optic disk, and/or blurringof the boundaries of the optic disk. The images of the retinas may betaken by any known and/or to be invented appropriate device.

In some embodiments, the abnormal medical event may involve the userhaving a migraine or another form of headache. With migraines and otherheadaches, vasoconstriction of facial or cranial blood vessels may leadto asymmetric changes in blood flow between the left and right sides ofthe head. Compensatory mechanisms may change smooth muscle constrictionaround blood vessels, further exacerbating this asymmetry. Thisvasoconstriction can lead to differential surface blood flow, musclecontraction, and facial temperature changes, leading to asymmetric bloodflow. As each individual's particular patterns of vasoconstriction wouldbe unique to the individual, the asymmetric phenomena may be differentfor different users. Thus, measuring deviation from the user's baselineblood flow patterns may increase the accuracy of detecting theseasymmetric phenomena, in some embodiments.

Additionally, the time course of migraine or headache usually involvesan occurrence over the course of minutes to hours (from the onset ofchanges to blood flow), and usually occurs with a characteristicpattern, allowing it to be differentiated from signs of other medical,artificial or external causes, which manifest different patterns ofblood flow and/or time courses.

In one embodiment, the abnormal medical event is a migraine attack, andthe deviation (of PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2)compared to a baseline based on the previous measurements of PPG_(SR1),PPG_(SR2), PPG_(SL1), and PPG_(SL2) of the user) is indicative of apattern of a certain change to facial blood flow, which is associatedwith a pattern of a change to facial blood flow of at least one previousmigraine attack, determined based on data comprising previous PPG_(SR1),PPG_(SR2), PPG_(SL1), and PPG_(SL2), which were measured starting fromat least 5 minutes before the previous migraine attack. In thisembodiment, the time of the beginning of the previous migraine attackcorresponds to the time at which the user became aware of the migraineattack.

In another embodiment, the abnormal medical event is headache, and thedeviation is indicative of at least one of: a change in directionalityof facial blood flow, and an asymmetrical reduction in blood flow to oneside of the face (for a period lasting more than one minute). For somepeople, a migraine attack and/or a headache may cause a change indirectionality of facial blood flow because the pulse propagation acrossthe face arrives at one side before it arrives to the other side. In oneexample, the changes in directionality of facial blood flow arecalculated from phase variations between PPG_(SR1), PPG_(SR2),PPG_(SL1), and PPG_(SL2) relative to a baseline for the user. For somepeople, vasoconstriction caused by a migraine attack and/or a headachemay affect the amplitude of the PPG signals, such as decreasing ofamplitudes of the PPG signals in a certain region. In one example, thereduction in blood flow to one side of the face is calculated fromchanges between amplitudes of PPG_(SR1), PPG_(SR2), PPG_(SL1), andPPG_(SL2) relative to the baseline.

In some embodiments, the abnormal medical event may involve the usersuffering from an infection. Inflammatory conditions, such ascellulitis, dermatitis and ear infection, originate in infection orinflammation in one particular region of the face, causing vasodilationleading to a facial asymmetry originating from phenomena such asincrease in swelling, redness and warmth, which are detectable only inthe vicinity of the infection. As each individual's baseline facialblood flow and coloration is different, comparing current measurementswith the baseline may allow accurate identification of vasodilationresulting from an inflammatory condition. The time course of suchinflammatory conditions would usually occur over the course of hours todays, allowing it to be differentiated from other medical or artificialphenomena, which may have similar signs but over a different timecourse.

Since inflammation often causes temperatures at the infected region torise, accuracy of detection of some abnormal medical events may increaseby measuring the temperature at different regions on the head. In oneembodiment, the system configured to detect an abnormal medical eventincludes right and left head-mounted thermometers, located at least 2 cmto the right and to the left of a vertical symmetry axis that dividesthe face, respectively. The head-mounted thermometers may be contactthermometers, such as thermistors, and/or non-contact thermal cameras.Optionally, the head-mounted thermometers measure ROIs on one or more ofthe following regions: the forehead, the temples, behind the ear, thecheeks, the nose, and the mouth. The right and left head-mountedthermometers take right and left temperature measurements, respectively.Optionally, the computer detects the abnormal medical event also basedon a deviation of the right and left temperature measurements from abaseline temperature for the user, where the baseline temperature forthe user is calculated based on data comprising previous right and lefttemperature measurements of the user, taken more than a day before theabnormal medical event. Optionally, the abnormal medical event detectedin this embodiment is selected from a set comprising cellulitis anddermatitis.

In one embodiments, the system configured to detect an abnormal medicalevent may optionally include right and left head-mounted thermometers,located less than 4 cm from the right and left earlobes, respectively,which provide right and left temperature measurements, respectively.Optionally, the computer detects the abnormal medical event also basedon a deviation of the right and left temperature measurements from abaseline temperature for the user, where the baseline temperature forthe user is calculated based on data comprising previous right and lefttemperature measurements of the user, taken more than a day before theabnormal medical event. In one example, the abnormal medical event isear infection. In another example, the abnormal medical event iscerebrovascular accident. In still another example, the abnormal medicalevent is mastoiditis.

Obtaining the PPG signals PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2)from measurements taken by the at least one right-side head-mounteddevice and the at least one left-side head-mounted device may involve,in some embodiments, performing various preprocessing operations inorder to assist in calculations and/or in extraction of the PPG signals.Optionally, the measurements may undergo various preprocessing stepsprior to being used by the computer to detect the abnormal medicalevent, and/or as part of the process of the detection of the abnormalmedical event. Some non-limiting examples of the preprocessing include:normalization of pixel intensities (e.g., to obtain a zero-mean unitvariance time series signal), and conditioning a time series signal byconstructing a square wave, a sine wave, or a user defined shape, suchas that obtained from an ECG signal or a PPG signal as described in U.S.Pat. No. 8,617,081.

In some embodiments, in which the at least one right-side head-mounteddevice and/or at least one left-side head-mounted device are cameras,images taken by the cameras may undergo various preprocessing to improvethe signal, such as color space transformation (e.g., transforming RGBimages into a monochromatic color or images in a different color space),blind source separation using algorithms such as independent componentanalysis (ICA) or principal component analysis (PCA), and variousfiltering techniques, such as detrending, bandpass filtering, and/orcontinuous wavelet transform (CWT). Various preprocessing techniquesknown in the art that may assist in extracting an PPG signals fromimages are discussed in Zaunseder et al. (2018), “Cardiovascularassessment by imaging photoplethysmography—a review”, BiomedicalEngineering 63(5), 617-634. An example of preprocessing that may be usedin some embodiments is given in U.S. Pat. No. 9,020,185, titled “Systemsand methods for non-contact heart rate sensing”, which describes how atimes-series signals obtained from video of a user can be filtered andprocessed to separate an underlying pulsing signal by, for example,using an ICA algorithm.

In some embodiments, detection of the abnormal medical event may involvecalculation of pulse arrival times (PATs) at one or more of the regionsROI_(R1), ROI_(R2), ROI_(L1) and ROI_(L2). Optionally, a PAT calculatedfrom an PPG signal represents a time at which the value representingblood volume (in the waveform represented in the PPG) begins to rise(signaling the arrival of the pulse). Alternatively, the PAT may becalculated as a different time, with respect to the pulse waveform, suchas the time at which a value representing blood volume reaches a maximumor a certain threshold, or the PAT may be the average of the time theblood volume is above a certain threshold. Another approach that may beutilized to calculate a PAT from an iPPG signal is described in Sola etal. “Parametric estimation of pulse arrival time: a robust approach topulse wave velocity”, Physiological measurement 30.7 (2009): 603, whichdescribe a family of PAT estimators based on the parametric modeling ofthe anacrotic phase of a pressure pulse.

Detection of the abnormal medical event may involve the computerutilizing an approach that may be characterized as involving machinelearning. In some embodiments, such a detection approach may involve thecomputer generating feature values based on data that includes PPGsignals (i.e., PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2) of theuser) and optionally other data, and then utilizing a previously trainedmodel to calculate one or more values indicative of whether the user isexperiencing the abnormal medical event (which may be any one of theexamples of values mentioned further above as being calculated by thecomputer for this purpose). It is to be noted that when the computercalculates a value that is indicative of the user having the abnormalmedical event, at least some of the feature values may reflect the factthat an asymmetrical change to blood flow had occurred. Optionally, theadditional data used to generate at least some of the feature valuesincludes previous measurements of PPG_(SR1), PPG_(SR2), PPG_(SL1), andPPG_(SL2) of the user, and/or measurements from additional sensorsand/or data sources as discussed below.

Feature values generated based on PPG signals may include various typesof values, which may be indicative of dynamics of the blood flow at therespective regions to which the PPG signals correspond. Optionally,these feature values may relate to properties of a pulse waveform, whichmay be a specific pulse waveform (which corresponds to a certain beat ofthe heart), or a window of pulse waveforms (e.g., an average property ofpulse waveforms in a certain window of time). Some examples of featurevalues that may be generated based on a pulse waveform include: the areaunder the pulse waveform, the amplitude of the pulse waveform, aderivative and/or second derivative of the pulse waveform, a pulsewaveform shape, pulse waveform energy, and pulse transit time (to therespective ROI). Some additional examples of features may be indicativeone or more of the following: a magnitude of a systolic peak, amagnitude of a diastolic peak, duration of the systolic phase, andduration of the diastolic phase. Additional examples of feature valuesmay include properties of the cardiac activity, such as the heart rateand heart rate variability (as determined from the PPG signal).Additionally, some feature values may include values of otherphysiological signals that may be calculated based on PPG signals, suchas blood pressure and cardiac output.

It is to be noted that the aforementioned feature values may becalculated in various ways. In one example, some feature values arecalculated for each PPG signal individually. In another example, somefeature values are calculated after normalizing a PPG signal withrespect to previous measurements from the corresponding PPG device usedto measure the PPG signal. In other examples, at least some of thefeature values may be calculated based on an aggregation of multiple PPGsignals (e.g., different pixels/regions in images captured by an iPPGdevice), or by aggregating values from multiple contact PPG devices.

In some embodiments, at least some of the feature values may representcomparative values, which provide an indication of the difference inblood flow, and/or in some other property that may be derived from a PPGsignal, between various regions on the head. Optionally, such featurevalues may assist in detecting asymmetrical blood flow (and/or changesthereto). In one example, the feature values include a certain featurevalue indicative of a difference in maximal amplitudes of one or more ofthe following pairs of PPG signals: (i) PPG_(SR1) and PPG_(SR2), (ii)PPG_(SR1) and PPG_(SR2), and (iii) PPG_(SR1) and PPG_(SL2). In anotherexample, the feature values include a certain feature value indicativeof a difference in a pulse arrival time between the following pairs ofregions of interest: (i) ROI_(R1) and ROI_(R2), (ii) ROI_(R1) andROI_(L1), and (iii) ROI_(R1) and ROI_(L2).

In some embodiments, at least some of the feature values describeproperties of pulse waveforms (e.g., various types of feature valuesmentioned above), which are derived from the previous measurements ofPPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2) of the user. Optionally,these feature values may include various blood flow baselines for theuser, which correspond to a certain situation the user was in when theprevious measurements were taken.

In some embodiments, at least some of the feature values may be “raw” orminimally processed measurements of the at least one right-sidehead-mounted device and/or at least one left-side head-mounted device.In one example, at least some of the feature values may be valuesobtained from contact PPG devices. In another example, at least some ofthe feature values may be pixel values obtained by inward-facinghead-mounted cameras. Optionally, the pixel values may be provided asinput to functions in order to generate at feature values that arelow-level image-based features. Some examples of low-level features,which may be derived from images, include feature generated using Gaborfilters, local binary patterns (LBP) and their derivatives, algorithmssuch as SIFT and/or SURF (and their derivatives), image keypoints,histograms of oriented gradients (HOG) descriptors, and products ofstatistical procedures such independent component analysis (ICA),principal component analysis (PCA), or linear discriminant analysis(LDA). Optionally, one or more of the feature values may be derived frommultiple images taken at different times, such as volume local binarypatterns (VLBP), cuboids, and/or optical strain-based features. In oneexample, one or more of the feature values may represent a differencebetween values of pixels at one time t and values of other pixels at adifferent region at some other time t+x (which, for example, can helpdetect different arrival times of a pulse wave).

In some embodiments, at least some feature values may be generated basedon other data sources (in addition to PPG signals). In some examples, atleast some feature values may be generated based on other sensors, suchas movement sensors (which may be head-mounted, wrist-worn, or carriedby the user some other way), head-mounted thermal cameras (e.g., asmentioned above), or other sensors used to measure the user. In otherexamples, at least some feature values may be indicative ofenvironmental conditions, such as the temperature, humidity, and/orextent of illumination (e.g., as obtained utilizing an outward-facinghead-mounted camera). Additionally, some feature values may beindicative of physical characteristics of the user, such as age, sex,weight, Body Mass Index (BMI), skin tone, and other characteristicsand/or situations the user may be in (e.g., level of tiredness,consumptions of various substances, etc.)

Stress is a factor that can influence the diameter of the arteries, andthus influence calculated values that relate to the PPG signals and/orblood flow. In one embodiment, the computer receives a value indicativeof a stress level of the user, and generates at least one of the featurevalues based on the received value. Optionally, the value indicative ofthe stress level is obtained using a thermal camera. In one example, thesystem may include an inward-facing head-mounted thermal camera thattakes measurements of a periorbital region of the user, where themeasurements of a periorbital region of the user are indicative of thestress level of the user. In another example, the system includes aninward-facing head-mounted thermal camera that takes measurements of aregion on the forehead of the user, where the measurements of the regionon the forehead of the user are indicative of the stress level of theuser. In still another example, the system includes an inward-facinghead-mounted thermal camera that takes measurements of a region on thenose of the user, where the measurements of the region on the nose ofthe user are indicative of the stress level of the user.

Hydration is a factor that affects blood viscosity, which can affect thespeed at which the blood flows in the body. In one embodiment, thecomputer receives a value indicative of a hydration level of the user,and generates at least one of the feature values based on the receivedvalue. Optionally, the system includes an additional camera that detectsintensity of radiation that is reflected from a region of exposed skinof the user, where the radiation is in spectral wavelengths chosen to bepreferentially absorbed by tissue water. In one example, saidwavelengths are chosen from three primary bands of wavelengths ofapproximately 1100-1350 nm, approximately 1500-1800 nm, andapproximately 2000-2300 nm. Optionally, measurements of the additionalcamera are utilized by the computer as values indicative of thehydration level of the user.

The following are examples of embodiments that utilize additional inputsto generate feature values used to detect the abnormal medical event. Inone embodiment, the computer receives a value indicative of atemperature of the user's body, and generates at least one of thefeature values based on the received value. In another embodiment, thecomputer receives a value indicative of a movement of the user's body,and generates at least one of the feature values based on the receivedvalue. For example, the computer may receive the input form ahead-mounted Inertial Measurement Unit (IMU) that includes a combinationof accelerometers, gyroscopes, and optionally magnetometers, and/or anIMU in a mobile device carried by the user. In yet another embodiment,the computer receives a value indicative of an orientation of the user'shead, and generates at least one of the feature values based on thereceived value. For example, the computer may receive the valuesindicative of the head's orientation from an outward-facing head-mountedcamera, and/or from a nearby non-wearable video camera. In still anotherembodiment, the computer receives a value indicative of consumption of asubstance by the user, and generates at least one of the feature valuesbased on the received value. Optionally, the substance comprises avasodilator and/or a vasoconstrictor.

The model utilized to detect the abnormal medical event may begenerated, in some embodiments, based on data obtained from one or moreusers, corresponding to times in which the one or more users were notaffected by the abnormal medical event, and additional data obtainedwhile the abnormal medical event occurred and/or following that time.Thus, this training data may reflect PPG signals and/or blood flow bothat normal times, and changes to PPG signals and/or blood flow that mayensue due to the abnormal medical event. This data may be used togenerate samples, each sample including feature values generated basedon PPG signals of a user and optionally additional data (as describedabove), and a label. The PPG signals include PPG_(SR1), PPG_(SR2),PPG_(SL1), and PPG_(SL2) of the user at a certain time, and optionallyprevious PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2) of the user,taken before the certain time. The label is a value related to thestatus of the abnormal medical event. For example, the label may beindicative of whether the user, at the certain time, experienced theabnormal medical event. In another example, the label may be indicativeof the extent or severity of the abnormal medical event at the certaintime. In yet another example, the label may be indicative of theduration until an onset of the abnormal medical event. In still anotherexample, the label may be indicative of the duration that has elapsedsince the onset of the abnormal medical event.

In some embodiments, the model used by the computer to detect theabnormal medical event from measurements of a certain user may begenerated, at least in part, based on data that includes previousmeasurements of the certain user (and as such, may be consideredpersonalized to some extent for the certain user). Additionally oralternatively, in some embodiments, the model may be generated based ondata of other users. Optionally, the data used to train the model mayinclude data obtained from a diverse set of users (e.g., users ofdifferent ages, weights, sexes, preexisting medical conditions, etc.).Optionally, the data used to train the model which is used to detect theabnormal medical event with a certain user includes data of other userswith similar characteristics to the certain user (e.g., similar weight,age, sex, height, and/or preexisting condition).

In order to achieve a robust model, which may be useful for detectingthe abnormal medical event for a diverse set of conditions, in someembodiments, the samples used for the training of the model may includesamples based on data collected when users were in different conditions.Optionally, the samples are generated based on data collected ondifferent days, while indoors and outdoors, and while differentenvironmental conditions persisted. In one example, the model is trainedon samples generated from a first set of training data taken duringdaytime, and is also trained on other samples generated from a secondset of training data taken during nighttime. In a second example, themodel is trained on samples generated from a first set of training datataken while users were exercising and moving, and is also trained onother samples generated from a second set of data taken while users weresitting and not exercising.

Utilizing the model to detect the abnormal medical event may involve thecomputer performing various operations, depending on the type of model.The following are some examples of various possibilities for the modeland the type of calculations that may be accordingly performed by thecomputer, in some embodiments, in order to calculate a value indicativeof whether the user was experiencing the abnormal medical event: (a) themodel comprises parameters of a decision tree. Optionally, the computersimulates a traversal along a path in the decision tree, determiningwhich branches to take based on the feature values. A value indicativeof whether the user was experiencing the abnormal medical event may beobtained at the leaf node and/or based on calculations involving valueson nodes and/or edges along the path; (b) the model comprises parametersof a regression model (e.g., regression coefficients in a linearregression model or a logistic regression model). Optionally, thecomputer multiplies the feature values (which may be considered aregressor) with the parameters of the regression model in order toobtain the value indicative of whether the user was experiencing theabnormal medical event; and/or (c) the model comprises parameters of aneural network. For example, the parameters may include values definingat least the following: (i) an interconnection pattern between differentlayers of neurons, (ii) weights of the interconnections, and (iii)activation functions that convert each neuron's weighted input to itsoutput activation. Optionally, the computer provides the feature valuesas inputs to the neural network, computes the values of the variousactivation functions and propagates values between layers, and obtainsan output from the network, which is the value indicative of whether theuser was experiencing the abnormal medical event.

In some embodiments, a machine learning approach that may be applied tocalculating a value indicative of whether the user is experiencing anabnormal medical event may be characterized as “deep learning”. In oneembodiment, the model may include parameters describing multiple hiddenlayers of a neural network. Optionally, the model may include aconvolution neural network (CNN). In one example, the CNN may beutilized to identify certain patterns in video images, such as thepatterns of corresponding to blood volume effects andballistocardiographic effects of the cardiac pulse. Due to the fact thatcalculating a value indicative of whether the user is experiencing theabnormal medical event may be based on multiple, possibly successive,images that display a certain pattern of change over time (i.e., acrossmultiple frames), these calculations may involve retaining stateinformation that is based on previous images. Optionally, the model mayinclude parameters that describe an architecture that supports such acapability. In one example, the model may include parameters of arecurrent neural network (RNN), which is a connectionist model thatcaptures the dynamics of sequences of samples via cycles in thenetwork's nodes. This enables RNNs to retain a state that can representinformation from an arbitrarily long context window. In one example, theRNN may be implemented using a long short-term memory (LSTM)architecture. In another example, the RNN may be implemented using abidirectional recurrent neural network architecture (BRNN).

The following method for detecting an abnormal medical event may be usedby systems modeled according to FIG. 3a or FIG. 3b . The steps describedbelow may be performed by running a computer program having instructionsfor implementing the method. Optionally, the instructions may be storedon a computer-readable medium, which may optionally be a non-transitorycomputer-readable medium. In response to execution by a system includinga processor and memory, the instructions cause the system to perform thefollowing steps:

In Step 1, measuring, utilizing at least one right-side head-mounteddevice, at least two signals indicative of photoplethysmographic signals(PPG_(SR1) and PPG_(SR2), respectively) at first and second regions ofinterest (ROI_(R1) and ROI_(R2), respectively) on the right side of auser's head. ROI_(R1) and ROI_(R2) are located at least 2 cm apart.

In Step 2, measuring, utilizing at least one left-side head-mounteddevice, at least two signals indicative of photoplethysmographic signals(PPG_(SL1) and PPG_(SL2), respectively) at first and second regions ofinterest (ROI_(L1) and ROI_(L2), respectively) on the left side of theuser's head. ROI_(L1) and ROI_(L2) are located at least 2 cm apart.

And in Step 3, detecting the abnormal medical event based on anasymmetrical change to blood flow recognizable in PPG_(SR1), PPG_(SR2),PPG_(SL1), and PPG_(SL2), relative to a baseline based on previousmeasurements of PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2) of theuser, taken before the abnormal medical event.

In some embodiments, detecting the abnormal medical event is doneutilizing a machine learning-based approach. Optionally, the methodincludes the following steps: generating feature values based on datacomprising: (i) PPG_(SR1), PPG_(SR2), PPG_(SL1), and PPG_(SL2) of theuser, and (ii) the previous measurements of PPG_(SR1), PPG_(SR2),PPG_(SL1), and PPG_(SL2) of the user; and utilizing a model tocalculate, based on the feature values, a value indicative of whetherthe user is experiencing the abnormal medical event.

Various embodiments described herein involve detections of physiologicalresponses based on user measurements. Some examples of physiologicalresponses include stress, an allergic reaction, an asthma attack, astroke, congestive heart failure, dehydration, intoxication (includingdrunkenness), a headache (which includes a migraine), and/or fatigue.Other examples of physiological responses include manifestations offear, startle, sexual arousal, anxiety, joy, pain or guilt. Still otherexamples of physiological responses include physiological signals suchas cardiovascular parameters (such as heart rate, blood pressure, and/orcardiac output), temperature, values of eye-related parameters (such aseye movements and/or pupil diameter), values of speech relatedparameters (such as frequencies and/or tempo), and/or values ofrespiratory related parameters (such as respiration rate, tidal volume,and/or exhale duration) of the user. Optionally, detecting aphysiological response may involve one or more of the following:determining whether the user has/had the physiological response,identifying an imminent attack associated with the physiologicalresponse, and/or calculating the extent of the physiological response.

In some embodiments, detection of the physiological response is done byprocessing measurements that fall within a certain window of time thatcharacterizes the physiological response. For example, depending on thephysiological response, the window may be five seconds long, thirtyseconds long, two minutes long, five minutes long, fifteen minutes long,or one hour long. Detecting the physiological response may involveanalysis of measurements taken during multiple of the above-describedwindows, such as measurements taken during different days. In someembodiments, a computer may receive a stream of measurements, takenwhile the user wears an HMS with coupled cameras and/or other sensorsduring the day, and periodically evaluate measurements that fall withina sliding window of a certain size.

In some embodiments, models are generated based on measurements takenover long periods. Sentences of the form of “measurements taken duringdifferent days” or “measurements taken over more than a week” are notlimited to continuous measurements spanning the different days or overthe week, respectively. For example, “measurements taken over more thana week” may be taken by eyeglasses equipped with cameras and/or othersensors, which are worn for more than a week, 8 hours a day. In thisexample, the user is not required to wear the eyeglasses while sleepingin order to take measurements over more than a week. Similarly,sentences of the form of “measurements taken over more than 5 days, atleast 2 hours a day” refer to a set comprising at least 10 measurementstaken over 5 different days, where at least two measurements are takeneach day at times separated by at least two hours.

Utilizing measurements taken over a long period (e.g., measurementstaken on “different days”) may have an advantage, in some embodiments,of contributing to the generalizability of a trained model. Measurementstaken over the long period likely include measurements taken indifferent environments, and/or measurements taken while the measureduser was in various physiological and/or mental states (e.g.,before/after meals, and/or while the measured user wassleepy/energetic/happy/depressed, etc.). Training a model on such datacan improve the performance of systems that utilize the model in thediverse settings often encountered in real-world use (as opposed tocontrolled laboratory-like settings). Additionally, taking themeasurements over the long period may have the advantage of enablingcollection of a large amount of training data that is required for somemachine learning approaches (e.g., “deep learning”)

Detecting the physiological response may involve performing varioustypes of calculations by a computer. Optionally, detecting thephysiological response may involve performing one or more of thefollowing operations: comparing measurements to a threshold (when thethreshold is reached that may be indicative of an occurrence of thephysiological response), comparing measurements to a reference timeseries, and/or by performing calculations that involve a model trainedusing machine learning methods. Optionally, the measurements upon whichthe one or more operations are performed are taken during a window oftime of a certain length, which may optionally depend on the type ofphysiological response being detected. In one example, the window may beshorter than one or more of the following durations: five seconds,fifteen seconds, one minute, five minutes, thirty minutes, one hour,four hours, one day, or one week. In another example, the window may belonger than one or more of the aforementioned durations. Thus, whenmeasurements are taken over a long period, such as measurements takenover a period of more than a week, detection of the physiologicalresponse at a certain time may be done based on a subset of themeasurements that falls within a certain window near the certain time;the detection at the certain time does not necessarily involve utilizingall values collected throughout the long period.

In some embodiments, detecting the physiological response of a user mayinvolve utilizing baseline measurement values, most of which were takenwhen the user was not experiencing the physiological response.Optionally, detecting the physiological response may rely on observing achange to typical measurement value at one or more ROIs (the baseline),where different users might have different typical measurement values atthe ROIs (i.e., different baselines). Optionally, detecting thephysiological response may rely on observing a change to a baselinelevel, which is determined based on previous measurements taken duringthe preceding minutes and/or hours.

In some embodiments, detecting a physiological response involvesdetermining the extent of the physiological response, which may beexpressed in various ways that are indicative of the extent of thephysiological response, such as: (i) a binary value indicative ofwhether the user experienced, and/or is experiencing, the physiologicalresponse, (ii) a numerical value indicative of the magnitude of thephysiological response, (iii) a categorical value indicative of theseverity/extent of the physiological response, (iv) an expected changein measurements of an ROI, and/or (v) rate of change in measurements ofan ROI. Optionally, when the physiological response corresponds to aphysiological signal (e.g., a heart rate, a breathing rate, or an extentof frontal lobe brain activity), the extent of the physiologicalresponse may be interpreted as the value of the physiological signal.

One approach for detecting a physiological response, which may beutilized in some embodiments, involves comparing measurements of one ormore ROIs to a threshold. In these embodiments, the computer may detectthe physiological response by comparing the measurements, and/or valuesderived therefrom (e.g., a statistic of the measurements and/or afunction of the measurements), to the threshold to determine whether itis reached. Optionally, the threshold may include a threshold in thetime domain, a threshold in the frequency domain, an upper threshold,and/or a lower threshold. When a threshold involves a certain change toa value (such as temperature or heart rate), the certain change may bepositive or negative. Different physiological responses described hereinmay involve different types of thresholds, which may be an upperthreshold (where reaching the threshold means ≥the threshold) or a lowerthreshold (where reaching the threshold means ≤the threshold).

Another approach for detecting a physiological response, which may beutilized in some embodiments, may be applicable when the measurements ofa user are treated as time series data. In some embodiments, thecomputer may compare measurements (represented as a time series) to oneor more reference time series that correspond to periods of time inwhich the physiological response occurred. Additionally oralternatively, the computer may compare the measurements to otherreference time series corresponding to times in which the physiologicalresponse did not occur. Optionally, if the similarity between themeasurements and a reference time series corresponding to aphysiological response reaches a threshold, this is indicative of thefact that the measurements correspond to a period of time during whichthe user had the physiological response. Optionally, if the similaritybetween the measurements and a reference time series that does notcorrespond to a physiological response reaches another threshold, thisis indicative of the fact that the measurements correspond to a periodof time in which the user did not have the physiological response. Timeseries analysis may involve various forms of processing involvingsegmenting data, aligning data, clustering, time warping, and variousfunctions for determining similarity between sequences of time seriesdata. Some of the techniques that may be utilized in various embodimentsare described in Ding, Hui, et al. “Querying and mining of time seriesdata: experimental comparison of representations and distance measures.”Proceedings of the VLDB Endowment 1.2 (2008): 1542-1552, and in Wang,Xiaoyue, et al. “Experimental comparison of representation methods anddistance measures for time series data.” Data Mining and KnowledgeDiscovery 26.2 (2013): 275-309.

A user interface (UI) may be utilized, in some embodiments, to notifythe user and/or some other entity, such as a caregiver, about thephysiological response, and/or present an alert responsive to anindication that the extent of the physiological response reaches athreshold. The UI may include a screen to display the notificationand/or alert, a speaker to play an audio notification, a tactile UI,and/or a vibrating UI. In some embodiments, “alerting” about aphysiological response of a user refers to informing about one or moreof the following non-limiting examples: the occurrence of aphysiological response that the user does not usually have (e.g., astroke, intoxication, and/or dehydration), an imminent physiologicalresponse (e.g., an allergic reaction, an epilepsy attack, and/or amigraine), and an extent of the physiological response reaching athreshold (e.g., stress and/or blood pressure reaching a predeterminedlevel).

Due to the mostly symmetric nature of the human body, when the faceundergoes temperature changes, e.g., due to external factors such as thetemperature in the environment or internal factors such as anactivity-related rise in body temperature, the changes to the face aregenerally symmetric. That is, the temperature changes at a region ofinterest (ROI) on the left side of the face (e.g., the left side of theforehead) are usually similar to the temperature changes at thesymmetric ROI on the right side of the face (e.g., the right side of theforehead). However, when the temperature on the face changes in anasymmetric way, this can be indicative of various physiologicalresponses and/or undesirable phenomena. Some examples of phenomena thatmay be identified by detecting asymmetric thermal patterns (“thermalasymmetry”) on a user's face include some types of strokes. In the caseof stroke, often the decreased blood flow in certain regions of the head(due to the stroke) can cause a decrease in the cutaneous temperaturesnear those certain regions.

Some embodiments utilize head-mounted sensors in order to detect strokesigns that involve detectable changes in temperature and/or blood flowdue to a stroke event.

In some embodiments, “stroke symptoms” refers to changes tofunction/sensation reported by the patient. In some embodiments, “strokesigns” refers to objective changes observable by a human individualand/or a sensor. Detecting stroke symptoms and/or stroke signs may alsobe referred to herein as detecting “whether the user has suffered from astroke”. It is to be noted that a detection that the user has sufferedfrom a stroke may be interpreted, in some cases, as indicating thatthere is a higher than normal risk that the user has suffered from astroke, and that certain actions should be taken (such as furtherinvestigation of the user's state by other means or seeking medicalattention). Thus, in some embodiments, detection by the computer thatthe user has suffered from a stroke may serve as an initial step thattriggers further steps for diagnosing the user's condition and not adefinitive final step in diagnosing the user's state.

In some embodiments, a system configured to detect a stroke based onthermal measurements includes at least one inward-facing head-mountedthermal camera (CAM) and computer. The at least one CAM is/areconfigured to take thermal measurements of at least first and secondregions on the right and left sides of the head (TH_(R1) and TH_(L1),respectively) of a user. Optionally, the at least one CAM is locatedbelow the first and second regions, and does not occlude the first andsecond regions. The computer is configured to detect, based on TH_(R1)and TH_(L1), whether the user has suffered a stroke. Optionally,detecting whether the user has suffered from a stroke refers to a recentstroke event, such as an ischemic or hemorrhagic stroke event thatstarted a short while before TH_(R1) and TH_(L1) were taken, where “ashort while” may be a period between minutes and several hours beforeTH_(R1) and TH_(L1) were taken. In this scenario, detection that theuser has suffered from a stroke may enable an intervention that canreduce the permanent damage of the stroke. Additionally detectingwhether the user has suffered from a stroke may also refer, in someembodiments, to earlier stroke events, which occurred more than sixhours before TH_(R1) and TH_(L1) were taken.

The first and second regions of which measurements are taken may includeportions of various parts of the head. For example, in differentembodiments, the first and second regions may cover a portion of atleast one of the following pairs of regions: the right and left sides ofthe forehead, the right and left temples, the right and left cheeks, theright and left earlobes, behind the right and left ears, periorbitalareas around the right and left eyes, the area of the right and leftmastoid processes.

In one embodiment, each of the at least one CAM is physically coupled toa frame worn on a user's head, and is located less than 15 cm, 5 cm, or2 cm from the user's face. In another embodiment, the at least one CAMis physically coupled to a clip-on, and the clip-on comprises a bodyconfigured to be attached and detached, multiple times, from a frameconfigured to be worn on the user's head.

In one embodiment, due to the angle between the optical axis of acertain CAM from among the at least one CAM and the Frankfort horizontalplane, the Scheimpflug principle may be employed in order to capturesharper images with the certain CAM. For example, when the user wears aframe to which the certain CAM is coupled, the certain CAM has a certaintilt greater than 2° between its sensor and lens planes, in order tocapture the sharper images. Additional details regarding application ofthe Scheimpflug are provided herein further below.

The at least one CAM may be a single CAM, in one embodiment, such as asingle FPA that captures images that include a portion of both sides ofthe forehead. In this embodiment, TH_(R1) and TH_(L1) includemeasurements that cover portions of the left side and right side ofuser's forehead, respectively. Optionally, TH_(R1) and TH_(L1) may bemeasured by different subsets of pixels of the FPA. An example of asystem that includes a single thermal camera is illustrated in FIG. 5,which illustrates a user wearing glasses with a single thermal sensor605 that measures ROIs 606 a and 606 b on the right and left of theforehead, respectively. The user in the figure has already been affectedby a stroke, as is evident by the lower temperature detected on the leftside of the forehead and the drooping of the left side of the face.

In other embodiments, the at least one CAM may include two or more CAMs.Optionally, the at least one CAM includes two CAMs, denoted CAM1 andCAM2. Optionally, CAM1 and CAM2 are located at least 0.5 cm to the rightand to the left of the vertical symmetry axis that divides the face,respectively. Optionally, each of CAM1 and CAM2 weighs below 10 g, 5 g,or 1 g.

FIG. 6 illustrates one example of a system for detecting a stroke thatincludes at least CAM1 and CAM2 described above. The figure illustratesa user wearing a frame with CAM1 and CAM2 (562 and 563, respectively)coupled thereto, which measure ROIs on the right and left cheeks (ROIs560 and 561, respectively). The measurements indicate that the left sideof the face is colder than the right side of the face. Based on thesemeasurements, and possibly additional data, the system detects thestroke and issues an alert. Optionally, the user's facial expression isslightly distorted and asymmetric, and a video camera providesadditional data in the form of images that may help in detecting thestroke.

FIG. 19 to FIG. 22 illustrate HMSs that may be used to detect a strokethat include two or more CAMs. FIG. 19 illustrates inward-facinghead-mounted cameras 30 and 31 that measure regions 32 and 33 on theforehead, respectively. FIG. 22 illustrates (i) inward-facinghead-mounted cameras 91 and 92 that are mounted to protruding arms andmeasure regions 97 and 98 on the forehead, respectively, (ii)inward-facing head-mounted cameras 95 and 96, which are also mounted toprotruding arms, which measure regions 101 and 102 on the lower part ofthe face, respectively, and (iii) head-mounted cameras 93 and 94 thatmeasure regions on the periorbital areas 99 and 100, respectively.

Depending on the locations the at least first and second regions on theright and left sides of the head, CAM1 and CAM2 mentioned above may belocated in specific locations with respect to the face. In one example,CAM1 and CAM2 are located outside the exhale stream of the mouth and/orthe exhale streams of the nostrils. In another example, each of CAM1 andCAM2 is located less than 10 cm from the face and there are anglesgreater than 20° between the Frankfort horizontal plane and the opticalaxes of CAM1 and CAM2.

Measurements of additional regions on the head may be used to detectwhether the user suffered from a stroke. In one embodiment, the at leastone CAM is further configured to take thermal measurements of at leastthird and fourth regions on the right and left sides of the head(TH_(R2) and TH_(L2), respectively), and the computer is furtherconfigured to detect whether the user has suffered a stroke also basedon TH_(R2) and TH_(L2) (in addition to TH_(R1) and TH_(L1)). Optionally,the middles of the first and second regions are at least 1 cm above themiddles of the third and fourth regions, respectively. Optionally, thethird and fourth regions cover at least a portion of one of thefollowing pairs of regions (which is not covered by the first and secondregions): the right and left sides of the forehead, the right and lefttemples, the right and left cheeks, the right and left earlobes, behindthe right and left ears, periorbital areas around the right and lefteyes, the area of the right and left mastoid processes.

In one embodiment, the system optionally includes at least oneoutward-facing head-mounted thermal camera (CAM_(out)), which isconfigured to take thermal measurements of the environment (TH_(ENV)).Optionally, the computer is further configured to also utilize TH_(ENV)(in addition to TH_(R1) and TH_(L1)) to detect whether the user hassuffered a stroke.

There are various approaches that may be used by the computer, indifferent embodiments, to detect based on TH_(R1) and TH_(L1) whetherthe user has suffered a stroke. In some embodiments, detecting whetherthe user has suffered a stroke involves comparing TH_(R1) and TH_(L1)(referred to as current measurements) to previously taken thermalmeasurements (previous measurements), such as previously taken TH_(R1)and TH_(L1) of the user. Optionally, the previous measurements are takenat least fifteen minutes before the current measurements. Optionally,the previous measurements are taken at least one hour before the currentmeasurements. Optionally, the previous measurements are taken at leastone six hours before the current measurements. Optionally, the previousmeasurements are taken at least one day before the current measurements.

In some embodiments, the previous measurements are taken over a longperiod and are used to calculate baseline thermal values of regions ofthe face, and/or used to generate various models, which are used todetect a stroke, as discussed below. It is to be noted that discussionbelow regarding detection of the stroke based on TH_(R1) and TH_(L1) canbe generalized to involve thermal measurements of other regions (such asTH_(R2) and TH_(L2) discussed above).

In one embodiment, the computer calculates a magnitude of thermalasymmetry of the head of the user based on a difference between TH_(R1)and TH_(L1), and compares the magnitude to a threshold. Responsive tothe magnitude reaching the threshold, the computer detects that the userhas suffered a stroke. Optionally, the difference between TH_(R1) andTH_(L1) needs to have reached the threshold (i.e., equal the thresholdvalue or exceed it) for at least a predetermined minimal duration, suchas at least one minute, at least five minutes, at least fifteen minutes,or at least some other period that is greater than 30 minutes.Optionally, the threshold is calculated based on previous magnitudes ofthermal asymmetry of the head of the user, which were calculated basedon previously taken TH_(R1) and TH_(L1) of the user. Thus, thisthreshold may be set according to a baseline thermal asymmetry thatrepresents the typical difference in the temperatures of the first andsecond regions; a stroke may be detected when the thermal asymmetrybecomes significantly different from this baseline.

In another embodiment, the computer calculates feature values andutilizes a model to calculate, based on the feature values, a valueindicative of whether the user has suffered a stroke. At least some ofthe feature values are generated based on TH_(R1) and TH_(L1) of theuser; examples of feature values that may be generated are given in thediscussion regarding feature values that may be generated to detect aphysiological response (herein suffering from a stroke is considered atype of physiological response). Optionally, at least some featurevalues are generated based on additional sources of information (otherthan the at least one CAM), such as additional thermal cameras,additional sensors that measure physiological signals of the user (e.g.,heart rate or galvanic skin response), and/or additional sensors thatmeasure the environment. Optionally, one or more of the feature valuesare indicative of the extent of difference between TH_(R1) and TH_(L1)of the user and previous TH_(R1) and TH_(L1) of the user, taken at leasta certain period earlier (such as at least fifteen minutes earlier, onehour earlier, a day earlier, or more than a day earlier). Thus, theseone or more feature values may represent a difference between thecurrent thermal measurements and baseline thermal values for the firstand second regions. Optionally, the model is generated based on datathat includes previously taken TH_(R1) and TH_(L1) of the user and/orother users. Optionally, when data includes previously taken TH_(R1) andTH_(L1) of other users, at least some of the measurements of the otherusers were taken while they did not suffer from a stroke, and at leastsome of the measurements of the other users were taken while they didsuffer from a stroke.

In another embodiment, the computer calculates a value indicative of ajoint probability of TH_(R1) and TH_(L1) based on a model that includesdistribution parameters calculated based on previously taken TH_(R1) andTH_(L1) of the user. Thus, the model describes a typical distribution ofthermal measurements on regions of the user's head (when the user didnot suffer from a stroke). The computer may compare the value indicativeof the joint probability to a threshold, and responsive to the valuebeing below the threshold, the computer detects that the user hassuffered a stroke. The threshold may represent an atypical thermalasymmetry, with very low probability to normally occur, which warrantsan alert of the possibility that the user has had a stroke.

In some embodiments described herein, detecting whether the user hassuffered a stroke may involve calculating a change to thermal asymmetryon the head based on a change between thermal measurements taken atdifferent times. This calculation can be performed in different ways, asdescribed below.

In one embodiment, the computer calculates the change between thethermal measurements as follows: calculate a temperature differencebetween the first and second regions at time x (ΔT_(X)) based on[TH_(R1) and TH_(L1)] taken at time x, calculate a temperaturedifference between the first and second regions at time y (ΔT_(y)) basedon [TH_(R1) and TH_(L1)] taken at time y, and calculate the outputindicative of the change in the thermal asymmetry on the face based on adifference between ΔT_(x) and ΔT_(y).

The embodiment described above may optionally be implemented using adifferential amplifier that receives TH_(R1) and TH_(L1) as inputs, andoutputs the temperature difference between the first and second regions.Optionally, the at least one CAM is/are based on thermopile sensors.Alternatively, the at least one CAM is/are based on pyroelectricsensors. In one example, pairs of thermal sensor elements are wired asopposite inputs to a differential amplifier in order for the thermalmeasurements to cancel each other and thereby remove the averagetemperature of the field of view from the electrical signal. This allowsthe at least one CAM to be less prone to providing false indications oftemperature changes in the event of being exposed to brief flashes ofradiation or field-wide illumination. This embodiment may also minimizecommon-mode interference, and as a result improve the accuracy of thethermal cameras.

In another embodiment, the computer calculates the change between thethermal measurements as follows: the computer calculates a temperaturechange between TH_(R1) taken at times t₁ and t₂ (ΔTH_(R1)), calculates atemperature change between TH_(L1) taken at times t₁ and t₂ (ΔTH_(L1)),and then calculates the output indicative of the thermal asymmetry onthe face based on a difference between ΔTH_(R1) and ΔTH_(L1).

It is noted that sentences such as “calculate a difference between X andY” or “detect a difference between X and Y” may be achieved by anyfunction that is proportional to the difference between X and Y.

The detection of a stroke based on TH_(R1) on TH_(L1), and optionallyother inputs, such as additional thermal measurements or additionalphysiological signals, may be sufficient, in some embodiments, to promptthe computer to take an action such as alerting the user, a caregiver,and/or emergency services. In other embodiments, the detection may beconsidered an indication that there is a risk that the user has suffereda stroke, and the computer may encourage the user to take a test (e.g.,the FAST test described below or portions thereof), in order to validatethe detection. Optionally, measurements and/or results taken during thetest may be used in addition to TH_(R1) and TH_(L1) and the optionaladditional inputs, or instead of that data, in order to make a second,more accurate detection of whether the user has suffered from a stroke.

Upon detecting that the user has suffered from a stroke, in someembodiments, the computer may prompt the user to take a test to validateand/or increase the confidence in the detection. This test may involveperforming one or more steps of a FAST test. Herein, FAST is an acronymused as a mnemonic to help detect and enhance responsiveness to theneeds of a person having a stroke. The acronym stands for Facialdrooping, Arm weakness, Speech difficulties and Time to call emergencyservices. Facial drooping involves a section of the face, usually onlyon one side, that is drooping and hard to move (often recognized by acrooked smile). Arm weakness typically involves an inability to raiseone's arm fully. Speech difficulties typically involve an inability ordifficulty to understand or produce speech.

FIG. 7 to FIG. 10 illustrate physiological and behavioral changes thatmay occur following a stroke, which may be detected using embodimentsdescribed herein. FIG. 7 illustrates a person at time zero, at the verybeginning of the onset of the stroke. At this time there may be nodetectable signs of the stroke. FIG. 8 illustrates the person's stateafter 5 minutes since the beginning of the stroke. By this time, certainchanges to the blood flow in the head, which were caused by the stroke,may be detectable. For example, the blood flow on one side of the facemay decrease. FIG. 9 illustrates the person's state 15 minutes after thebeginning of the stroke. At this time, temperature changes to regions ofthe face, which are due to the changes in blood flow, may be detectable.FIG. 10 illustrates the person's state 45 minutes after the beginning ofthe stroke. At this time, droopiness of one side of the face (e.g., dueto reduced blood flow and impaired neurologic control) may beobservable.

The following are some examples of steps that may be taken to furtherdiagnose whether the user has suffered from a stroke (e.g., steps from aFAST test). Optionally, results of these steps may be used to furtherstrengthen the computer's detection (e.g., to increase confidence in thedetection) or change the detection (e.g., and decide the detection was afalse alarm). Optionally, upon determining, based on one or more of thesteps, that the user may have a suffered a stroke, the computer performsat least one of the following steps: (i) instruct the user to seekemergency medical assistance, (ii) connect the user through live videochat with a medical specialist, and (iii) automatically alert apredetermined person and/or entity about the user's condition.

In one embodiment, a camera (e.g., a cellphone camera or aninward-facing camera coupled to a frame worn by the user) takes imagesof at least a portion of the user's face. Optionally, the computer isfurther configured to (i) instruct the user, via a user interface, tosmile and/or stick out the tongue, and (ii) detect, based on analysis ofthe images taken by the camera, whether a portion of one side of theuser's face droops and/or whether the user was able to stick out thetongue. Detecting that the portion of the one side of the user's facedroops may be done based on a comparison of the images of at least aportion of the user's face to previously taken images of at least theportion of the user's face, which were taken at least one hour before.Optionally, the computer utilizes a machine learning based model todetermine whether the comparison indicates that the face droops. In thiscase, the model may be trained based on sets of images that includebefore and after images of people who suffered a stroke. Additionally oralternatively, detecting that the portion of the one side of the user'sface droops is done using a machine learning-based model trained onexamples of images of faces of people with a portion of one side of theface drooping. Optionally, detecting whether the tongue is sticking outmay be done using image analysis and/or machine learning approaches(e.g., utilizing a model trained on previous images of the user and/orother users sticking out their tongue). FIG. 11 illustrates a user whois requested by a smartphone app to smile. Images of the user may betaken by the smartphone and analyzed in order to determine whether theuser has smiled and/or to what extent the smile is considered “normal”.Similarly, FIG. 12 illustrates a user who is requested by a smartphoneapp to stick out his tongue.

In another embodiment, a microphone is used to record the user's speech.The computer is configured to (i) instruct the user, via a userinterface, to speak (e.g., say a predetermined phrased), and (ii)detect, based on analysis of a recording of the user taken by themicrophone, whether the user's speech is slurred, and/or whether adifference between the recording and previous recordings of the userexceeds a threshold that indicates excessively slurred or difficult.Optionally, the computer uses a model trained based on examples ofpeoples' normal and incoherent speech (e.g., recordings of people beforeand after they had a stroke). FIG. 13 illustrates a user who isrequested by a smartphone app to say a sentence. The user's speech canbe recorded by the smartphone and analyzed to detect slurry and/oruncharacteristic speech.

In yet another embodiment, a sensor is used to take measurementsindicative of at least one of movement and position, of at least one ofthe user's arms. Optionally, the computer is further configured to (i)instruct the user, via a user interface, to raise at least one the arms,and (ii) detect, based on analysis of the measurements taken by thesensor, whether an arm of the user drifts downward. In one example, thesensor is in a cellphone held by the user. In another example, thesensor is in a smartwatch or bracelet on the user's wrist. In stillanother example, the sensor is embedded in a garment worn by the user.FIG. 14 illustrates a user who is requested by a smartphone app to raisehis arms. The user's movements can be detected by the smartphone inorder to determine whether one arm falls. In the illustrated example,the user may be requested to raise the arms at least twice, each timeholding the phone in a different hand. Such measurements can serve as abaseline to compare the rate of ascent and descent of each of the arms.

In still another embodiment, a sensor is used to take measurementsindicative of how the user walks. Optionally, the computer is furtherconfigured to (i) instruct the user, via a user interface, to walk, and(ii) detect, based on analysis of the measurements taken by the sensor,whether a difference between the measurements taken by the sensor andprevious measurements of the user taken by the sensor exceeds athreshold. Optionally, exceeding the threshold is indicative of a suddenloss of balance or coordination. In one example, the sensor is coupledto a frame worn on the user's head (e.g., a glasses frame). In anotherexample, the sensor is in a cellphone, smartwatch, bracelet, garment, orshoe worn by the user.

Early detection of a stroke, e.g., using one or more of the approachesdescribed above, can be essential for minimizing the stroke-relateddamage. With strokes, it is certainly the case that time is of theessence. Early administration of treatment (e.g., within a few hours)can in many cases lead to minimal long term stroke-related damage.However, following a certain window of time (e.g., six hours since thestart of the stroke), severe and irreversible long term damage mayoccur; impacting both quality of life and cost of care. FIG. 15 and FIG.16 illustrate the difference between a timely intervention andintervention that comes too late. In FIG. 15, a patient arrives a shortwhile (about an hour and fifteen minutes) after beginning of a stroke.This patient has moderate stroke damage (denoted by reference numeral655). Intervention at this early stage is mostly successful, and aftertwo months the patient has only slight long term damage (denoted byreference numeral 656), which involves a relatively small impact on thequality of life and a relatively low cost in terms of future medicalexpenses. FIG. 16 illustrates a case in which the patient arrives toolate (e.g., six hours after the start of the stroke). In this scenario,the patient already has more severe stroke damage (denoted by referencenumeral 657). Intervention at this late stage is mostly unsuccessful.Two months later, there is severe long term stroke-related damage, whichinvolves a relatively large impact on the quality of life and arelatively high cost in terms of future medical expenses.

In addition to detecting a stroke, some embodiments may be used todetect an occurrence of a migraine attack. In one embodiment, the firstand second regions are located above the user's eye level, and thecomputer is further configured to detect occurrence of a migraine attackbased on asymmetry between TH_(R1) and TH_(L1). In another embodiment,the first and second regions are located above the user's eye level. Thecomputer is further configured to generate feature values based onTH_(R1) and TH_(L1), and to utilize a model to calculate, based on thefeature values, a value indicative of whether the user is experiencing amigraine attack or whether a migraine attack is imminent. Optionally,the model if generated based on previous TH_(R1) and TH_(L1) of the usertaken while the user had a migraine attack or taken 30 minutes or lessbefore the user had a migraine attack.

In some embodiments, a system configured to detect a stroke based onasymmetric changes to blood flow includes at least first and seconddevices and a computer. The first and second devices are located to theright and to the left of the vertical symmetry axis that divides theface of a user, respectively; the first and second devices areconfigured to measure first and second signals (S_(BF1) and S_(BF2),respectively) indicative of blood flow in regions to right and to theleft of the vertical symmetry axis. The computer detects whether theuser has suffered a stroke based on an asymmetric change to blood flow,which is recognizable in S_(BF1) and S_(BF2). Optionally, the systemincludes a frame configured to be worn on a user's head, and the firstand second devices are head-mounted devices that are physically coupledto the right and left sides of the frame, respectively.

The first and second devices may be devices of various types.Optionally, the first and second devices are the same type of device.Alternatively, the first and second devices may be different types ofdevices. The following are some examples of various types of devicesthat can be used in embodiments described herein in order to measure asignal indicative of blood flow in a region of the body of the user.

In one embodiment, the first and second devices comprise first andsecond cameras, respectively. The first and second cameras are based onsensors comprising at least 3×3 pixels configured to detectelectromagnetic radiation having wavelengths in at least a portion ofthe range of 200 nm to 1200 nm. Optionally, the asymmetric changes toblood flow lead to asymmetric facial skin color changes to the face ofthe user. Optionally, the system includes first and second active lightsources configured to illuminate the first and second regions,respectively. In one example, the first and second devices and first andsecond active light sources are head-mounted, and the first and secondactive light sources are configured to illuminate the first and secondregions with electromagnetic radiation having wavelengths in at least aportion of the range of 800 nm to 1200 nm.

In addition to the first and second cameras mentioned above, someembodiments may include first and second outward-facing cameras thattake images of the environment to the right and left of the user's head,respectively. Optionally, the computer utilizes the images to detectwhether the user has suffered a stroke. Optionally, the images of theenvironment are indicative of illumination towards the face, imagestaken by the inward-facing cameras are indicative of reflections fromthe face. Thus, the images of the environment may be used to account, atleast in part, for variations in ambient light that may cause errors indetections of blood flow.

In another embodiment, the first and second devices function as imagingphotoplethysmography devices, and/or the first and second devicesfunction as pulse oximeters.

There are various types of devices that may be used in embodimentsdescribed herein, and ways in which the devices may be attached to theuser and/or located the user's proximity. In one example, the first andsecond device may be head-mounted devices, such as devices physicallycoupled to a frame worn on the user's head (e.g., a frame ofsmartglasses, such as augmented reality glasses, virtual realityglasses, or mixed reality glasses). In another example, the first andsecond devices are embedded in a garment worn by the user (e.g., a smartshirt) or some other wearable accessory (e.g., a necklace, bracelets,etc.). In yet another, the first and second devices may be located inheadrests of a chair in which the user sits. In still another example,the first and second devices are attached to walls (e.g., of a vehiclecabin in which the user sits or a room in which the user spends time).In yet another example, the first and second devices may be coupled torobotic arms. Optionally, the robotic arms may be used to move and/ororient the devices in order to account for the user's movements, whichmay enable the first and second devices to measure the same regions onthe user's head despite the user's movement.

The first and second devices may be utilized to measure various regionson the right and left sides of the user's head. In one embodiment, theregions on the right and left sides include portions of the right andleft temples, respectively. In another embodiment, the regions on theright and left sides include portions of the right and left sides of theforehead, respectively. Optionally, the center of the region on theright is at least 3 cm from the center of the region on the left. Instill another embodiment, the regions on the right and left sidesinclude portions of the right and left cheeks, respectively. In stillanother embodiment, the regions on the right and left sides includeareas behind the right and left ears, respectively. Optionally, theareas behind the right and left ears are below the hairline and cover atleast a portion of the mastoid process in their respective sides of thehead.

FIG. 17a illustrates an embodiment of a system that includes multiplepairs of right and left cameras, and locations on the face that they maybe used to measure. The cameras illustrated in the figure are coupled toa frame worn by the user, which may be, for example, any of the camerasmentioned herein. The illustrated example include: (i) cameras 641 a and641 b that are coupled to the right and left sides of the top of theframe, respectively, which measure ROIs 644 a and 644 b on the right andleft of the forehead, respectively; (ii) cameras 642 a and 642 b thatare coupled to the right and left sides of the bottom of the frame,respectively, which measure ROIs 645 a and 645 b on the right and leftcheeks, respectively; and (iii) cameras 643 a and 643 b that are coupledto the frame near the right and left sides of the nose, respectively,which measure ROIs 646 a and 646 b in the right and left periorbitalregions, respectively.

FIG. 17b illustrates a stroke sign 647, which involves decreased bloodflow in the forehead, and may be detected using the system illustratedin FIG. 17a . Another stroke sign that may be detected by the systemillustrated in FIG. 17c is stroke sign 648, which involves decreasedtemperature in the periorbital region.

FIG. 18a illustrates a system in which the first and second devices arehead-mounted cameras that are located behind the ears. The figureillustrates cameras 651, which may be a downward-facing video camerathat takes images that include portions of the side of the neck. FIG.18b illustrates a variant of the system illustrated in FIG. 18a in whichcamera 652 is a downward-facing camera that includes an extender bodywhich enables the camera to move beyond the hairline in order to obtainunobstructed images of the side of the neck.

FIG. 18c illustrates ischemic stroke 650, which restricts the blood flowto the side of the head (illustrated as patch 653), which may bedetected based on measurements of device 652 in order to alert about theuser having suffered from a stroke.

Various types of values, which are indicative of blood flow (and takenby the first or second devices), can be determined based on a signal,which is denoted S_(BF) in the discussion below. S_(BF) may be, forexample, aforementioned S_(BF1) or S_(BF2). Propagation of a cardiacpulse wave can lead to changes in the blood flow, with increased bloodflow as a result of blood pumped during the systole, and lower bloodflow at other times (e.g., during diastole). The changes in blood flowoften manifest as changes to values in S_(BF). For example, higher bloodflow may correspond to increased intensity of certain colors of pixels(when S_(BF) includes images), temperatures of pixels (when S_(BF)includes thermal images), or increased absorption of certain wavelengths(when S_(BF) includes measurements of a photoplethysmography device).

In some embodiments, calculating a statistic of S_(BF) may provide anindication of the extent of the blood flow. For example, the statisticmay be an average value (e.g., average pixel values), which is acalculated over a period, such as ten seconds, thirty seconds, or aminute. For example, in an embodiment in which the devices are camerasand S_(BF) includes images, the average hue of pixels in the images canbe indicative of the blood flow (e.g., a redder hue may correspond tomore intense blood flow). In an embodiment in which the devices arethermal sensors and S_(BF) includes thermal images, the averagetemperature of pixels in the thermal images can be indicative of theblood flow (e.g., a higher average temperature may correspond to moreintense blood flow).

In other embodiments, analysis of propagation of cardiac pulse waves, asmanifested in the values of S_(BF), can be used to provide indicationsof the extent of blood flow. When S_(BF) includes multiple pixels values(e.g., images or thermal images), propagation of a pulse wave typicallyinvolves an increase in pixel values that correspond to a period of ahigh blood flow during the pulse wave (e.g., when blood flow is drivenby a systole) followed by a decrease in pixel values that correspond toa period of a lower blood flow during the pulse wave (e.g., decreasingtowards a diastolic trough). The speed at which the pulse wavepropagates along a segment of the images of S_(BF) is indicative of thespeed of blood flow in the segment depicted in the images. For example,the speed at which a peak (e.g., corresponding to the systole) is seento propagate in the images is directly correlated with the blood flow.

Another parameter indicative of blood flow that can be derived fromanalysis of a propagation of a pulse wave, as it is manifested in valuesof S_(BF), is the amplitude of the signal. For example, signals thatdisplay a periodic increase and decrease in values of S_(BF), whichcorresponds to the heart rate, can be analyzed to determine thedifference in values (e.g., difference in color, temperature, orintensity) between the peak and the trough observed in the values ofS_(BF) during the transition of a pulse wave. In some embodiments, theamplitude of the signal (e.g., the difference between the value at thepeak and the value at the trough) is correlated with the blood flow. Inone example, the higher the amplitude of the signal, the higher theblood flow.

Another parameter indicative of blood flow that can be derived fromanalysis of a propagation of a pulse wave, as it is manifested in valuesof S_(BF), is the extent to which pixels in images (or thermal images)display a periodic change to their values that reaches a certainthreshold. For example, the area on the face in which the periodicfacial skin color changes (which correspond to the cardiac pulse) reacha certain threshold is correlated with the blood flow. The higher theblood flow, the larger the area.

The extent of Blood flow (e.g., the speed at which the blood flows) iscorrelated with the times at which a pulse wave is detected at differentlocations of the body. Typically, the farther a location from the heart,the later the pulse wave is detected (e.g., detection of a pulse wavemay correspond to the time at which a systolic peak is observed). Thisphenomenon can be utilized, in some embodiments, to calculate valuesindicative of blood flow based on the difference in times at which pulsewaves are detected at different locations. For example, analysis of oneor more S_(BF) signals, which include values that are indicative ofpropagation of a pulse wave at various locations, can determine thedifference in time (Δt) between detection of a pulse wave at differentlocations. The value of Δt is typically correlated with the blood flow:the higher the blood flow, the smaller Δt is expected to be.

The computer is configured, in some embodiments, to detect whether theuser has suffered a stroke based on an asymmetric change to blood flow,which is recognizable in S_(BF1) and S_(BF2).

Herein, sentences of the form “asymmetric change to blood flowrecognizable in S_(BF1) and S_(BF2)” refer to effects of blood flow thatmay be identified and/or utilized by the computer, which are usually notrecognized by the naked eye (in the case of images), but arerecognizable algorithmically when the signal values are analyzed. Forexample, blood flow may cause facial skin color changes (FSCC) thatcorresponds to different concentrations of oxidized hemoglobin due tovarying blood pressure caused by the cardiac pulse. Similar blood flowdependent effects may be viewed with other types of signals (e.g.,slight changes in cutaneous temperatures due to the flow of blood.

When the blood flow on both sides of the head and/or body are monitored,asymmetric changes may be recognized. These changes are typicallydifferent from symmetric changes that can be caused by factors such asphysical activity (which typically effects the blood flow on both sidesin the same way). An asymmetric change to the blood flow can mean thatone side has been affected by an event, such as a stroke, which does notinfluence the other side. In one example, the asymmetric change to bloodflow comprises a change in blood flow velocity on left side of the facethat is 10% greater or 10% lower than a change in blood flow velocity onone right side of the face. In another example, the asymmetric change toblood flow comprises a change in the volume of blood the flows during acertain period in the left side of the face that is 10% greater or 10%lower than the volume of blood that flows during the certain period inthe right side of the face. In yet another example, the asymmetricchange to blood flow comprises a change in the direction of the bloodflow on one side of the face (e.g., as a result of a stroke), which isnot observed at the symmetric location on the other side of the face.

In some embodiments, the computer detects whether the user has suffereda stroke based on a comparison between values belonging to a current setof S_(BF1) and S_(BF2) and values belonging to a previous set of S_(BF1)and S_(BF2) of the user. Optionally, the previous set was measured atleast fifteen minutes before the current set, at least one hour beforethe current set, or at least one day before the current set. Optionally,the previous set serves to establish baseline blood flow values for bothregions when the user was assumed not to be effected by a stroke.

In one embodiment, the computer uses a machine learning model to detectwhether the user has suffered a stroke. Optionally, the computer isconfigured to: generate feature values based on data comprising acurrent set of S_(BF1) and S_(BF2) and a previous set of S_(BF1) andS_(BF2) of the user, and utilize the model to calculate, based on thefeature values, a value indicative of whether the user has suffered astroke. Optionally, the previous set was measured at least one hourbefore the current set. Optionally, at least some of the feature valuesare indicative of changes to the blood flow in each of the first andsecond regions between the time the current set was measured and whenthe previous set was measured.

In one embodiment, the model was generated based on data comprising“after” sets of S_(BF1) and S_(BF2) of other users and corresponding“before” sets of S_(BF1) and S_(BF2) of the other users. In theseembodiments, the “after” sets were measured after the other users hadsuffered from a stroke (and the “before” sets were measured before theysuffered from the stroke). Thus, training samples generated based onthis data can reflect some of the types of asymmetric changes to bloodflow that characterize suffering from a stroke.

In the event that a stroke is detected, the computer may prompt the userto take a FAST test (or portions thereof), as described above. Inanother example, the computer may suggest to the user to take images ofthe retinas. In this example, the computer is further configured tocompare the images of the retinas with previously taken images of theretinas of the user, and to detect whether the user has suffered astroke based on the comparison. Optionally, the comparison can take intoaccount diameter of retinal arteries, swelling and blurring of theboundaries of the optic disk.

Blood flow usually exhibits typical patterns in the user's body. When ablood flow pattern changes, this usually happens in a predictable way(e.g., increase in blood flow due to physical activity, certainemotional responses, etc.). When a person's blood flow changes in anatypical way, this may indicate an occurrence of a certain medicalincident, such as a stroke. Therefore, atypical blood flow patternsshould be detected in order to investigate their cause.

In some embodiments, a system configured to detect an atypical bloodflow pattern in the head of a user includes one or more head-mounteddevices and a computer. The one or more head-mounted devices areconfigured to measure at least three signals (S_(BF1), S_(BF2) andS_(BF3), respectively), indicative of blood flow in at least threecorresponding regions of interest on the head (ROI₁, ROI₂, and ROI₃,respectively) of a user. Optionally, the centers of ROI₁, ROI₂ and ROI₃are at least 1 cm away from each other. Optionally, the one or morehead-mounted devices do not occlude ROI₁, ROI₂ and ROI₃.

There are various types of devices the one or more head-mounted devicesmay be. Optionally, each of the one or more head-mounted devices are thesame type of device. Alternatively, when the one or more head-mounteddevices are a plurality of devices, the plurality of devices may be ofdifferent types of devices. The following are some examples of varioustypes of devices that can be used in embodiments described herein inorder to measure a signal indicative of blood flow in a region of thebody of the user.

In one embodiment, the one or more head-mounted devices include a camerathat is based on a sensor comprising at least 3×3 pixels configured todetect electromagnetic radiation having wavelengths in at least aportion of the range of 200 nm to 1200 nm. Optionally, the systemincludes one or more light sources configured to ROI₁, ROI₂ and ROI₃.Optionally, the one or more light sources are configured to illuminateROI₁, ROI₂ and ROI₃ with electromagnetic radiation having wavelengths inat least a portion of the range of 800 nm to 1200 nm.

In another embodiment, the one or more head-mounted devices include adevice that functions as an imaging photoplethysmography device and/or adevice that functions as a pulse oximeter.

The computer is configured, in one embodiment, to generate a blood flowpattern based on a current set of S_(BF1), S_(BF2) and S_(BF3). Thecomputer is also configured to calculate, based on a set of previousblood flow patterns of the user, a value indicative of the extent towhich the blood flow pattern is atypical. Optionally, the set ofprevious blood flow patterns were generated based on sets of S_(BF1),S_(BF2) and S_(BF3) measured at least one day prior to when the currentset was measured. That is, the previous sets of blood flow patterns weregenerated based on data the comprises S_(BF1), S_(BF2) and S_(BF3)measured at least one day before S_(BF1), S_(BF2) and S_(BF3) in thecurrent set. Optionally, the previous sets of blood flow patterns weregenerated based on data that was measured when the user was not known tobe in an atypical condition (e.g., the user was considered healthy atthe time).

In some embodiments, the value indicative of the extent to which theblood flow pattern is atypical is compared to a threshold. If the valuereaches the threshold, the computer may take different actions. In oneembodiment, responsive to the value reaching the threshold, the computerissues an alert. For example, the computer may send a message to apredetermined recipient (e.g., emergency services or a caregiver). Inanother example, the computer may command a user interface to generatean alert (e.g., a beeping sound, a text message, or vibrations. Inanother embodiment, responsive to the value reaching the threshold, thecomputer may prompt the user to perform a test, using an electronicdevice, to determine whether the user has suffered a stroke. Forexample, the user may be prompted to perform a FAST test (or portionsthereof), as described elsewhere in this disclosure.

There are various ways in which a blood flow pattern may be generatedbased on S_(BF1), S_(BF2) and S_(BF3). In one embodiment, S_(BF1),S_(BF2) and S_(BF3) themselves may constitute the pattern. Optionally,the blood flow pattern comprises a time series that is indicative ofblood flow at each of ROI₁, ROI₂, and ROI₃. In one example, the bloodflow pattern may be a time series of the raw (unprocessed) values ofS_(BF1), S_(BF2) and S_(BF3) (e.g., a time series of values measured bythe one or more head-mounted devices). In another example, blood flowpattern may be a time series of processed values, such as variousstatistics of S_(BF1), S_(BF2) and S_(BF3) at different points of time(e.g., average pixel intensity for images taken by a camera at differentpoints of time).

In another embodiment, the blood flow pattern may include statistics ofthe values of S_(BF1), S_(BF2) and S_(BF3), such as values representingthe average blood flow during the period S_(BF1), S_(BF2) and S_(BF3)were measured. Values in a blood flow pattern, such as the statistics ofthe values of S_(BF1), S_(BF2) and S_(BF3) may be of various types ofvalues. The following are some examples of types of values that may beused in a “blood flow pattern”. In one example, the blood flow patternincludes one or more values that describe the intensity of blood flow ateach of ROI₁, ROI₂, and/or ROI₃. In another example, the blood flowpattern includes one or more values that describe the amplitude ofchanges to measurement values at ROI″, ROI₂, and/or ROI₃, such as theamplitude of periodic changes (which correspond to the heart rate) tocolor, temperature, and/or absorbance at certain wavelengths. In yetanother example, the blood flow pattern includes one or more values thatdescribe the speed at which a pulse wave propagates through ROI₁, ROI₂,and/or ROI₃. In still another example, the blood flow pattern includesone or more values that describe the direction at which a pulse wavepropagates in ROI₁, ROI₂, and/or ROI₃.

There are various ways in which the computer may utilize the previousblood flow patterns in order to determine the extent to which the bloodflow pattern is atypical.

In one embodiment, the computer calculates the value indicative of theextent to which the blood flow pattern is atypical by calculatingdistances between the blood flow pattern and each of the previous bloodflow patterns, and determining a minimal value among the distances. Thelarger this minimal value, the more atypical the blood flow pattern maybe considered (due to its difference from all previous blood flowpatterns considered). In one example, in which blood flow patternsinclude time series data, the distances may be calculated by finding theextent of similarity between the blood flow pattern and each of theprevious blood flow patterns (e.g., using methods described in Wang,Xiaoyue, et al. “Experimental comparison of representation methods anddistance measures for time series data”, Data Mining and KnowledgeDiscovery 26.2 (2013): 275-309). In another example, distances betweenblood flow patterns that include dimensional data, such as blood flowpatterns that may be represented as vectors of values, may be calculatedusing various similarity metrics known in the art such as Euclideandistances or vector dot products.

In another embodiment, the computer calculates, based on the previoussets of S_(BF1), S_(BF2) and S_(BF3), parameters of a probabilitydensity function (pdf) for blood flow patterns. For example, each bloodflow pattern may be represented as a vector of values and the pdf may bea multivariate distribution (e.g., a multivariate Gaussian) whoseparameters are calculated based on vectors of values representing theprevious blood flow patterns. Given a vector of values representing theblood flow pattern, the probability of the vector of values iscalculated based on the parameters of the pdf. Optionally, if theprobability is below a threshold, the blood flow pattern may beconsidered atypical. Optionally, the threshold is determined based onprobabilities calculated for the vectors of values representing theprevious blood flow patterns, such the at least a certain proportion ofthe vectors have a probability that reaches the threshold. For example,the certain proportion may be at least 75%, at least 90%, at least 99%,or 100%.

In yet another embodiment, the computer calculates the value indicativeof the extent to which the blood flow pattern is atypical using a modelof a one-class classifier generated based on the set of previous bloodflow patterns.

In still another embodiment, the computer calculates feature values andutilizes a model to calculate, based on the feature values, the valueindicative of the extent to which the blood flow pattern is atypical.Optionally, one or more of the feature values are generated based on theblood flow pattern, and at least one of the feature values is generatedbased on the previous blood flow patterns. Optionally, feature valuesgenerated based on a blood flow pattern include one or more of thevalues described in examples given above as examples of values in ablood flow pattern. Optionally, one or more of the feature valuesdescribe differences between values representing the blood flow pattern,and values representing the previous blood flow patterns. The model isgenerated based on samples, with each sample comprising: (i) featurevalues calculated based on a blood flow pattern of a certain user, fromamong a plurality of users, and previous blood flow patterns of thecertain user, and (ii) a label indicative of an extent to which theblood flow pattern of the certain user is atypical. Additional detailsregarding generating the model can be found herein in the discussionregarding machine learning approaches that may be used to detect aphysiological response.

In some embodiments, the computer may use additional inputs to determinewhether the blood flow pattern is an atypical blood flow pattern. In oneexample, the computer may receive measurements of various physiologicalsignals (e.g., heart rate, respiration, or brain activity) and use thesemeasurements to generate at least some of the feature values. In anotherexample, the computer may receive an indication of a state of the userand to generate one or more of the feature values based on theindication. Optionally, the state of the user is indicative of at leastone of the following: an extent of physical activity of the user, andconsuming a certain substance by the user (e.g., alcohol or a drug).

The physiological and emotional state of a person can often beassociated with certain cortical activity. Various phenomena, which maybe considered abnormal states, such as anger or displaying symptomaticbehavior of Attention Deficit Disorder (ADD) or Attention DeficitHyperactivity Disorder (ADHD), are often associated with certainatypical cortical activity. This atypical cortical activity can changethe blood flow patterns on the face, and especially on the foreheadarea. Thus, there is a need for a way to detect such changes in bloodflow in real world day-to-day situations. Preferably, in order to becomfortable and more aesthetically acceptable, these measurements shouldbe taken without involving direct physical contact with the forehead oroccluding it.

In some embodiments, a system configured to detect an attackcharacterized by an atypical blood flow pattern includes the one or morehead-mounted devices (described above) and a computer. The one or morehead-mounted devices are configured to measure at least three signals(S_(BF1), S_(BF2) and S_(BF3), respectively), indicative of blood flowin at least three corresponding regions of interest on the head (ROI₁,ROI₂, and ROI₃, respectively) of a user. Optionally, the centers ofROI₁, ROI₂ and ROI₃ are at least 1 cm away from each other. Optionally,the one or more head-mounted devices do not occlude ROI₁, ROI₂ and ROI₃.Optionally, ROI₁, ROI₂ and ROI₃ are on the same side of the face.Alternatively, ROI₁, ROI₂ and ROI₃ are on all on the same side of theface. For example, ROI₁ may be on the left side of the face, and ROI₂ onthe right side of the face.

The computer is configured, in one embodiment, to calculate a value,based on S_(BF1), S_(BF2), and S_(BF3), indicative of whether the useris in a normal or abnormal state.

In one embodiment, the state of the user is determined by comparing ablood flow pattern generated based on S_(BF1), S_(BF2), and S_(BF3) toreference blood flow patterns of the user that include at least onereference blood flow pattern that corresponds to the normal state and atleast one reference blood flow pattern that corresponds to the abnormalstate. Optionally, a reference blood flow pattern is determined fromprevious S_(BF1), S_(BF2), and S_(BF3) of the user, taken while the userwas in a certain state corresponding to the reference blood flow pattern(e.g., normal or abnormal states). Optionally, if the similarity reachesa threshold, the user is considered to be in the state to which thereference blood flow pattern corresponds.

In another embodiment, the computer determines that the user is in acertain state (e.g., normal or abnormal) by generating feature values(at least some of which are generated based on S_(BF1), S_(BF2), andS_(BF3)) and utilizing a model to calculate, based on the featurevalues, the value indicative of whether the user is in a normal orabnormal state. Optionally, the model is trained based on samples, eachcomprising feature values generated based on previous S_(BF1), S_(BF2),and S_(BF3) of the user, taken while the user was in the certain state.

In yet another embodiment, S_(BF1), S_(BF2), and S_(BF3) comprise timeseries data, and the computer calculates the value indicative of whetherthe user is in a normal or abnormal state based on comparing the timeseries to at least first and second reference time series, eachgenerated based on previously taken S_(BF1), S_(BF2), and S_(BF3) of theuser; the first reference time series is based on previous S_(BF1),S_(BF2), and S_(BF3) taken while the user was in a normal state, and thesecond reference time series is based on previous S_(BF1), S_(BF2), andS_(BF3) taken while the user was in an abnormal state. Optionally, thetime series data and/or the first and second reference time seriesincludes data taken over at least a certain period of time (e.g., atleast ten seconds, at least one minute, or at least ten minutes).

Being in a normal/abnormal state may correspond to different behavioraland/or physiological responses. In one embodiment, the abnormal stateinvolves the user displaying symptoms of one or more of the following:an anger attack, Attention Deficit Disorder (ADD), and Attention DeficitHyperactivity Disorder (ADHD). In this embodiment, being in the normalstate refers to usual behavior of the user that does not involvedisplaying said symptoms. In another embodiment, when the user is in theabnormal state, the user will display within a predetermined duration(e.g., shorter than an hour), with a probability above a predeterminedthreshold, symptoms of one or more of the following: anger, ADD, andADHD. In this embodiment, when the user is in the normal state, the userwill display the symptoms within the predetermined duration with aprobability below the predetermined threshold. In yet anotherembodiment, when the user is in the abnormal state the user suffers froma headache and/or migraine (or an onset of a migraine attack will occuris a short time such as less than one hour), and when the user is in thenormal state, the user does not suffer from a headache and/or amigraine. In still another embodiment, the abnormal state refers totimes in which the user has a higher level of concentration compared tothe normal state that refers to time in which the user has a usual levelof concentration. Although the blood flow patterns of the forehead areusually specific to the user, they are usually repetitive, and thus thesystem may able to learn some blood flow patterns of the user thatcorrespond to various states.

Determining the user's state based on S_(BF1), S_(BF2), and S_(BF3) (andoptionally other sources of data) may be done using a machinelearning-based model. Optionally, the model is trained based on samplescomprising feature values generated based on previous S_(BF1), S_(BF2),and S_(BF3) taken when the user was in a known state (e.g., fordifferent times it was known whether the user was in the normal orabnormal state). Optionally, the user may provide indications abouthis/her state at the time, such as by entering values via an app whenhaving a headache, migraine, or an anger attack. Additionally oralternatively, an observer of the user, which may be another person or asoftware agent, may provide the indications about the user's state. Forexample, a parent may determine that certain behavior patterns of achild correspond to displaying symptomatic behavior of ADHD. In anotherexample, indications of the state of the user may be determined based onmeasurements of physiological signals of the user, such as measurementsof the heart rate, heart rate variability, breathing rate, galvanic skinresponse, and/or brain activity (e.g., using EEG). Optionally, thevarious indications described above are used to generate labels for thesamples generated based on the previous S_(BF1), S_(BF2), and S_(BF3).

In some embodiments, one or more of the feature values in the samplesmay be based on other sources of data (different from S_(BF1), S_(BF2),and S_(BF3)). These may include additional physiological measurements ofthe user and/or measurements of the environment in which the user waswhile S_(BF1), S_(BF2), and S_(BF3) were taken. In one example, at leastsome of the feature values used in samples include additionalphysiological measurements indicative of one or more of the followingsignals of the user: heart rate, heart rate variability, brainwaveactivity, galvanic skin response, muscle activity, and extent ofmovement. In another example, at least some of the feature values usedin samples include measurements of the environment that are indicativeof one or more of the following values of the environment in which theuser was in: temperature, humidity level, noise level, air quality, windspeed, and infrared radiation level.

Given a set of samples comprising feature values generated based onS_(BF1), S_(BF2), and S_(BF3) (and optionally the other sources of data)and labels generated based on the indications, the model can be trainedusing various machine learning-based training algorithms. Optionally,the model is utilized by a classifier that classifies the user's state(e.g., normal/abnormal) based on feature values generated based onS_(BF1), S_(BF2), and S_(BF3) (and optionally the other sources).

The model may include various types of parameters, depending on the typeof training algorithm utilized to generate the model. For example, themodel may include parameters of one or more of the following: aregression model, a support vector machine, a neural network, agraphical model, a decision tree, a random forest, and other models ofother types of machine learning classification and/or predictionapproaches.

In some embodiments, the model is trained utilizing deep learningalgorithms. Optionally, the model includes parameters describingmultiple hidden layers of a neural network. Optionally, the modelincludes a convolution neural network (CNN), which is useful foridentifying certain patterns in images, such as patterns of colorchanges on the forehead. Optionally, the model may be utilized toidentify a progression of a state of the user (e.g., a gradual formingof a certain blood flow pattern on the forehead). In such cases, themodel may include parameters that describe an architecture that supportsa capability of retaining state information. In one example, the modelmay include parameters of a recurrent neural network (RNN), which is aconnectionist model that captures the dynamics of sequences of samplesvia cycles in the network's nodes. This enables RNNs to retain a statethat can represent information from an arbitrarily long context window.In one example, the RNN may be implemented using a long short-termmemory (LSTM) architecture. In another example, the RNN may beimplemented using a bidirectional recurrent neural network architecture(BRNN).

In order to generate a model suitable for identifying the state of theuser in real-world day-to-day situations, in some embodiments, thesamples used to train the model are based on S_(BF1), S_(BF2), andS_(BF3) (and optionally the other sources of data) taken while the userwas in different situations, locations, and/or conducting differentactivities. In a first example, the model may be trained based on afirst set of previous S_(BF1), S_(BF2), and S_(BF3) taken while the userwas indoors and in the normal state, a second set of previous S_(BF1),S_(BF2), and S_(BF3) taken while the user was indoors and in theabnormal state, a third set of previous S_(BF1), S_(BF2), and S_(BF3)taken while the user was outdoors and in the normal state, and a fourthset of previous S_(BF1), S_(BF2), and S_(BF3) taken while the user wasoutdoors and in the abnormal state. In a second example, the model maybe trained based on a first set of previous S_(BF1), S_(BF2), andS_(BF3) taken while the user was sitting and in the normal state, asecond set of previous S_(BF1), S_(BF2), and S_(BF3) taken while theuser was sitting and in the abnormal state, a third set of previousS_(BF1), S_(BF2), and S_(BF3) taken while the user was standing and/ormoving around and in the normal state, and a fourth set of previousS_(BF1), S_(BF2), and S_(BF3) taken while the user was standing and/ormoving around and in the abnormal state. Usually the movements whilestanding and/or moving around, and especially when walking or running,are greater compared to the movement while sitting; therefore, a modeltrained on samples taken during both sitting and standing and/or movingaround is expected to perform better compared to a model trained onsamples taken only while sitting.

Having the ability to determine the state of the user can beadvantageous when it comes to scheduling tasks for the user and/ormaking recommendations for the user, which suits the user's state. Inone embodiment, responsive to determining that the user is in the normalstate, the computer prioritizes a first activity over a second activity,and responsive to determining that the user is in the abnormal state,the computer prioritizes the second activity over the first activity.Optionally, accomplishing each of the first and second activitiesrequires at least a minute of the user's attention, and the secondactivity is more suitable for the abnormal state than the firstactivity. Optionally, and the first activity is more suitable for thenormal state than the second activity. Optionally, prioritizing thefirst and second activities is performed by a calendar managementprogram, a project management program, and/or a “to do” list program.Optionally, prioritizing a certain activity over another means one ormore of the following: suggesting the certain activity before suggestingthe other activity, suggesting the certain activity more frequently thanthe other activity (in the context of the specific state), allottingmore time for the certain activity than for the other activity, andgiving a more prominent reminder for the certain activity than for theother activity (e.g., an auditory indication vs. a mention in a calendarprogram that is visible only if the calendar program is opened).

Such state-dependent prioritization may be implemented in variousscenarios. In one example, the normal state refers to a normalconcentration level, the abnormal state refers to a lower than normalconcentration level, and the first activity requires a high attentionlevel from the user compared to the second activity. For instance, thefirst and second activities may relate to different topics of aself-learning program for school; when identifying that the user is inthe normal concentration state, a math class is prioritized higher thana sports lesson; and when identifying that the user is in the lowerconcentration state, the math class is prioritized lower than the sportslesson. In another example, the normal state refers to a normal angerlevel, the abnormal state refers to a higher than normal anger level,and the first activity involves more interactions of the user with otherhumans compared to the second activity. In still another example, thenormal state refers to a normal fear level, the abnormal state refers toa panic attack, and the second activity is expected to have a morerelaxing effect on the user compared to the first activity.

Passively taken sensor based measurements may be used, in someembodiments, in order to detect whether a user exhibits stroke signs.However, in some scenarios, sensors that may be used to detect strokesigns may provide inaccurate signals that can lead to a high rate offalse alarms. Detecting whether a user exhibits stroke signs can also bedone by an app that prompts the user to perform one or more activitiesinvolved in a FAST test, and analyzing the user's actions whileperforming the one or more activities. However, apps for detectingstroke signs are often not used because the person suffering from astroke is not aware of the incident (and thus does not initiate a test).Thus, the combination of the two approaches can increase the rate atwhich stroke signs may be detected based on possibly inaccuratemeasurements, by having the sensor measurements serve as a trigger toactivate an app to prompt the user to perform the one or more activitiesinvolved in the FAST test.

In one embodiment, a system configured detect stroke signs includes atleast first, second, and third sensors, and a computer.

The first and second sensors configured to take first and secondmeasurements (M_(R) and M_(L), respectively) of regions belonging to theright and left sides of a user's body. Optionally, M_(R) and M_(L) areindicative of blood flow in the regions on the right and left sides ofthe user's body, respectively. The following are examples of varioustypes of sensors that may be used, in some embodiments, as the first andsecond sensors.

In one example, the first and second sensors are electroencephalography(EEG) electrodes that measure brain activity and are positioned on theright and left sides of the head, respectively. Optionally, the firstand second sensors are EEG electrodes implanted under the user's scalp.

In another example, the first and second sensors are electromyography(EMG) sensors implanted in the left and right sides of the user's body,respectively. Alternatively, the first and second sensors may be EMGsensors attached to the surface of the user's body.

In yet another example, the first and second sensors may be ultrasoundsensors. Optionally, the ultrasound sensors are positioned such thatthey are in contact with the surface of the right and left sides of theuser's body respectively.

In still another example, the first and second sensors arephotoplethysmogram (PPG) sensors that provide measurements indicative ofthe blood flow on the right and left sides of the user's body,respectively.

In yet another example, the first and second sensors comprise camerasbased on a sensor comprising at least 3×3 pixels configured to detectelectromagnetic radiation having wavelengths in at least a portion ofthe range of 200 nm to 1200 nm.

In yet another example, the first and second sensors are thermistors incontact with the right and left sides of the user's body. For example,the first and second sensors may be embedded in one or more of thefollowing: a clothing item worn by the user, gloves, or a scarf.

And in still another example, the first and second sensors are thermalcameras each comprising at least 3×3 pixels configured to detectelectromagnetic radiation having wavelengths above 2500 nm.

The computer calculates, based on M_(R) and M_(L), a value indicative ofa risk that the user has suffered from a stroke. Responsive todetermining that the value reaches a threshold, the computer mayinstruct the user, via a user interface, to perform the predeterminedactivity. The computer may then detect whether the user exhibits strokesigns based on measurements (M_(A)) of a third sensor, which were takenwhile the user performed the predetermined activity. Optionally, thecomputer utilizes M_(R) and M_(L) along with M_(A) in order to make amore accurate detection of stroke signs. Optionally, detection based onM_(R), M_(L), and M_(A) is more accurate than detection based on M_(R)alone M_(L).

Calculating the value indicative of the risk that the user has sufferedfrom a stroke may be done in different ways in different embodiments. Inone embodiment, the value indicative of the risk is indicativedifference between of extents of physiological changes in the right andleft sides of the body, which are calculated based on M_(R) and M_(L),respectively. For example, the value may be indicative of a differencein blood flow between the right and left sides of the body, a differencein the temperature between the right and left sides, and/or a differencein skin color and/or extent of changes to skin color between the rightand left sides. Optionally, when the difference reaches the threshold,this means that the user is at a risk of having suffered a stroke thatis sufficiently high to warrant an additional test that involvesperforming the predetermined activity and measuring the user with thethird sensor.

In another embodiment, the computer generates feature values based onM_(R) and M_(L) and utilizes a model to calculate, based on the featurevalues, the value indicative of the risk that the user has suffered froma stroke. Optionally, the model is generated based on previous M_(R) andM_(L) of multiple users. For example, the model may be generated basedon data that comprises M_(R) and M_(L) of users who were healthy at thetime the measurements were taken, and also based on M_(R) and M_(L) ofusers who suffered from a stroke while the measurements were taken.

There are various predetermined activities the computer may prompt theuser to perform in order to detect whether the user exhibits strokesigns. These activities may be measured using different sensors. Thefollowing are examples of combinations of activities, and sensors thatmay be used to take the measurements M_(A) of the user while the userperforms the activities.

In one embodiment, performing the predetermined activity comprisessmiling. Optionally, in this embodiment, the third sensor is a camera.For example, the camera may be a camera embedded in a cell phone (orsome other device) held by the user or some other person. In anotherexample, the camera may be coupled to a frame worn on the user's head.In this embodiment, M_(A) include images of the user's face that capturethe user's facial expressions while the user was instructed to smile.The computer may detect the stroke signs based on analysis of images ofthe user which are indicative of one side of the face drooping.Optionally, the analysis involves comparing M_(A) to previously takenimages of the user's face.

In another embodiment, performing the predetermined activity comprisesraising both arms. Optionally, in this embodiment, the third sensor isan inertial measurement unit (IMU). For example, the IMU may be embeddedin a cell phone, a smart watch, or another device on worn or held by theuser (e.g., on the hand, wrist, or arm). In this embodiment, thecomputer detects the stroke signs based on analysis of measurements ofmovements of the user (taken by the IMU) which are indicative of one armdrifting downward.

In yet another embodiment, performing the predetermined activitycomprises walking. Optionally, in this embodiment, the third sensor isan inertial measurement unit (IMU). For example, the IMU may be coupledto a frame worn on the user's head or in a device carried by the user(e.g., a smartphone or a smartwatch). In this embodiment, computerdetects the stroke signs based on analysis measurements of movements ofthe user (taken by the IMU) which are indicative of imbalance of theuser.

In still another embodiment, performing the predetermined activitycomprises repeating a phrase. Optionally, in this embodiment, the thirdsensor is a microphone. For example, the microphone may belong to adevice carried or worn by the user (e.g., a smartphone, a smartwatch, ormicrophone embedded in a head-mounted display). In this embodiment,computer detects the stroke signs based on analysis of a recording ofvoice of the user which is indicative of the user's speech being slurry.Optionally, the analysis of the recording of the user's voice is donebased on a comparison with previous recordings of the user's voice.

In some embodiments, the computer may conduct tests to determine whetherthe user's brain function has been affected. For example, the computermay prompt the user to perform various tasks using an app that can testfor memory and cognitive skills, such as answering trivia questions,solving puzzles, etc.

In some embodiments, following the user's performance of thepredetermined activity and analysis of M_(A), results of the analysismay be used to improve the accuracy of the calculation of the risk. Forexample, responsive to determining, based on specific M_(A), that theuser does not exhibit stroke signs after measuring specific M_(R) andM_(L) values (which indicated that there is sufficient risk), thecomputer is configured to adjust the threshold based on the specificM_(R) and M_(L) values. In another example, the computer may utilize thespecific M_(R) and M_(L) to retrain a model used to calculate the risk.In this example, the specific M_(R) and M_(L) can constitute an exampleof a false positive that is used to adjust the model parameters.

One of the signs to having a stroke is sudden loss of balance orcoordination. It is easier to identify loss of balance or coordinationfrom measurements taken by a head-mounted IMU compared to measurementstaken by an IMU inside a wrist band or a phone in the hand. The reasonis that the movements of the head include much less noise compared tomovements of a hand that is used to manipulate items. Therefore, ahead-mounted IMU can be more beneficial for identifying loss of balanceor coordination than a wrist-mounted IMU.

In one embodiment, a system configured to detect a medical condition,which involves a loss of at least one of balance and coordination,includes at least a head-mounted inertial measurement unit (IMU) and acomputer. Some examples of medical conditions that involve loss ofbalance and/or coordination include a stroke and a spell of dizziness.

The IMU is configured to measure movements of the head of a user wearingthe IMU, and a computer. Optionally, the IMU comprises at least one ofthe following elements: an accelerometer, a gyroscope, and amagnetometer. Optionally, the IMU is coupled to a frame worn on theuser's head.

The computer is configured to detect the medical condition based onmeasurements of the IMU (M_(current)) and previous measurements ofmovements of the head of the user wearing the IMU (M_(previous)).Optionally, at least some of M_(previous) were taken while the user didnot have the medical condition. Optionally, the computer is furtherconfigured to alert a predetermined recipient (e.g., a caregiver oremergency services) responsive to detecting the medical condition.

There are various ways in which the medical condition may be detectedbased on M_(current) and M_(previous). In one embodiment, the computeris configured to detect the medical condition by calculating adifference between movement characteristics determined based onM_(current) and typical movement characteristics of the user calculatedbased on M_(previous). Optionally, if the difference reaches athreshold, the user is considered to have the medical condition.

In another embodiment, the computer is configured to detect the medicalcondition by generating feature values and utilizing a model tocalculate a value, based on the feature values, which is indicative ofwhether the user has the medical condition. Optionally, one or more ofthe feature values are calculated based on M_(current) and the model isgenerated based M_(previous) of the user.

In yet another embodiment, the computer is configured to detect themedical condition by generating feature values and utilizing a model tocalculate a value, based on the feature values, which is indicative ofwhether the user has the medical condition. Optionally, one or more ofthe feature values are calculated based on movements of the head of oneor more other users wearing an IMU.

The user's movement characteristics may be affected by various factors,as discussed below. Thus, receiving an indication of such factors thatinfluence movement and balance can help improve accuracy of detectingthe medical condition based on M_(current) and M_(previous). In someembodiments, the computer is further configured to receive an indicationof a situation in which the user is in while M_(current) are taken, andto utilize the indication to detect the medical condition.

In one embodiment, the situation comprises being under influence of amedication, and the indication of the situation is received from atleast one of: a pill dispenser, a sensor-enabled pill, and a usertracking application. Examples of user tracking applications include:(i) software that requires a user to log events, such as consumption ofa certain item, usage of a certain item, having a certain experience,and/or being in a certain situation, (ii) an image analysis softwarethat receives images taken by the user or a third party nearby the user,and infers the situation from the images using image processingtechniques. Examples of sources for the image include: smart-glasseswith outfacing camera, augmented reality devices, mobile phones, andwebcams.

In another one embodiment, the situation comprises being under influenceof at least one of alcohol and caffeine, and the indication of thesituation is received from at least one of: a refrigerator, a pantry, aserving robot, and a user tracking application; and wherein theindication indicates that the user took an alcoholic beverage.

In yet another embodiment, the situation comprises being under a certainstress level, and the indication of the situation is received from asensor that measures a physiological signal of the user, such as athermal sensor, a heart rate sensor, a sensor of galvanic skin response,or an EEG sensor.

In still another embodiment, the situation is selected from a groupcomprising: being inside a moving vehicle, and being on a static floor;and the indication of the situation is received from a positioningsystem. Examples of positioning systems include: GPS, wirelesspositioning systems, image processing to infer position.

Typically having a stroke will influence brain activity, which can bedetected through electroencephalography (EEG). However, this signal mayat times be noisy and insufficient to reliably detect a stroke. Thus,additional tests may be needed.

In one embodiment, a system configured to detect stroke signs based onelectroencephalography (EEG) includes at least a sensor configured totake measurements (M_(A)) of the user, and a computer.

The computer is configured to: receive EEG signals of the user, and toutilize a model to calculate, based on the EEG signals, a valueindicative of a risk that the user has suffered from a stroke. The valueindicative of the risk is compared by the computer to a threshold, andresponsive to determining that the value reaches a threshold, thecomputer instructs the user, via a user interface, to perform apredetermined activity. The computer then detects whether the user hasstroke signs based on M_(A) taken while the user performed thepredetermined activity. Optionally, the computer utilizes the EEGsignals along with M_(A) in order to make a more accurate detection ofstroke signs. Optionally, detection based on the EEG signals and M_(A)is more accurate than detection based on the EEG signals alone.

Calculating the value indicative of the risk that the user has sufferedfrom a stroke may be done in different ways in different embodiments. Inone embodiment, the value indicative of the risk is indicativedifference between of extents of electrical potential changes in theright and left sides of the brain, which are calculated based on EEGsignals from electrodes on the right and left sides of the head.Optionally, when the difference reaches the threshold, this means thatthe user is at a risk of having suffered a stroke that is sufficientlyhigh to warrant an additional test that involves performing thepredetermined activity and measuring the user with the third sensor.

In another embodiment, the computer generates feature values based onthe EEG signals and utilizes a model to calculate, based on the featurevalues, the value indicative of the risk that the user has suffered froma stroke. Optionally, the model is generated based on previous EEGsignals of multiple users. For example, the model may be generated basedon data that comprises EEG signals of users who were healthy at the timethe measurements were taken, and also based on EEG signals of users whosuffered from a stroke while the measurements were taken.

There are various predetermined activities the computer may prompt theuser to perform in order to detect whether the user exhibits strokesigns. These activities may be measured using different sensors.Examples of combinations of activities, and sensors that may be used totake the measurements M_(A) of the user while the user performs theactivities are given above (e.g., smiling, raising hands, walking, andspeaking a phrase).

In some embodiments, following the user's performance of thepredetermined activity and analysis of M_(A) results of the analysis maybe used to improve the accuracy of the calculation of the risk. Forexample, responsive to determining, based on specific M_(A), that theuser does not exhibit stroke signs after measuring specific EEG signalsvalues (which indicated that there is sufficient risk), the computer isconfigured to adjust the threshold based on the specific EEG signals. Inanother example, the computer may utilize the specific EEG signals toretrain a model used to calculate the risk. In this example, thespecific EEG signals can constitute an example of a false positive thatis used to adjust the model parameters.

US Patent Application 2019/0223737A1, which is herein incorporated byreference in its entirety and is a previous patent application of theApplicant of this invention, discusses and illustrates in paragraphs0040-0049, together with their associated drawings, various examples ofhead-mounted systems equipped with head-mounted cameras, which can beadapted to be utilized with some of the embodiments herein. For example,these paragraphs illustrate various inward-facing head-mounted camerascoupled to an eyeglasses frame, illustrate cameras that capture regionson the periorbital areas, illustrate an optional computer that mayinclude a processor, memory, a battery and/or a communication module,illustrate inward-facing head-mounted cameras coupled to an augmentedreality devices, illustrate head-mounted cameras coupled to a virtualreality device, illustrate head-mounted cameras coupled to a sunglassesframe, illustrate cameras configured to capture various ROIs, such asthe forehead, the upper lip, the cheeks, and sides of the nose,illustrate inward-facing head-mounted cameras mounted to protrudingarms, illustrate various inward-facing head-mounted cameras havingmulti-pixel sensors (FPA sensors) configured to capture various ROIs,illustrate head-mounted cameras that are physically coupled to a frameusing a clip-on device configured to be attached/detached from a pair ofeyeglasses in order to secure/release the device to/from the eyeglasses,illustrate a clip-on device holds at least an inward-facing camera, aprocessor, a battery, and a wireless communication module, illustrateright and left clip-on devices configured to be attached behind aneyeglasses frame, illustrate a single-unit clip-on device configured tobe attached behind an eyeglasses frame, and illustrate right and leftclip-on devices configured to be attached/detached from an eyeglassesframe and having protruding arms to hold the inward-facing head-mountedcameras.

It is noted that the elliptic and other shapes of the ROIs in some ofthe drawings are just for illustration purposes, and the actual shapesof the ROIs are usually not as illustrated. It is possible to calculatethe accurate shape of an ROI using various methods, such as acomputerized simulation using a 3D model of the face and a model of ahead-mounted system (HMS) to which a camera is physically coupled,and/or by placing a LED instead of the sensor, while maintaining thesame field of view (FOV) and observing the illumination pattern on theface, and/or by analyzing the image captured by the camera (when itsresolution is high enough). Furthermore, illustrations and discussionsof a camera represent one or more cameras, where each camera may havethe same FOV and/or different FOVs. Unless indicated to the contrary,the cameras may include one or more sensing elements (pixels), even whenmultiple sensing elements do not explicitly appear in the figures orexplicitly mentioned in the text; when a camera includes multiplesensing elements then the illustrated ROI usually refers to the totalROI captured by the camera, which is made of multiple regions that arerespectively captured by the different sensing elements. The positionsof the cameras in the figures are just for illustration, and the camerasmay be placed at other positions on the HMS.

Sentences in the form of an “ROI on an area”, such as ROI on theforehead or an ROI on the nose, refer to at least a portion of the area.Depending on the context, and especially when using a camera having asmall number of pixels, the ROI may cover another area (in addition tothe area). For example, a sentence in the form of “an ROI on the nose”may refer to either: 100% of the ROI is on the nose, or some of the ROIis on the nose and some of the ROI is on the upper lip.

In various embodiments, cameras are located close to a user's face, suchas at most 2 cm, 5 cm, 10 cm, 15 cm, or 20 cm from the face. Thedistance from the face/head in sentences such as “a camera located lessthan 10 cm from the face/head” refers to the shortest possible distancebetween the camera and the face/head. The head-mounted cameras used invarious embodiments may be lightweight, such that each camera weighsbelow 10 g, 5 g, 1 g, and/or 0.5 g (herein “g” denotes to grams).

In some embodiments, a device, such as a camera, may be positioned suchthat it occludes an ROI on the user's face, while in other embodiments,the device may be positioned such that it does not occlude the ROI.Non-limiting examples of occluding and non-occluding systems, which canbe adapted to be utilized with some of the embodiments herein, aredescribed in paragraph 0034 and its associated drawings in US PatentApplication 2019/0223737A1.

Some of the embodiments herein may utilize the Scheimpflug principle,which is a known geometric rule that describes the orientation of theplane of focus of a camera when the lens plane is tilted relative to thesensor plane. US Patent Application 2019/0223737A1 discusses andillustrates in paragraphs 0097-0105, together with their associateddrawings, various embodiments of the Scheimpflug principle, which can beadapted to be utilized with some of the embodiments herein. For example,these paragraphs discuss selecting the tilt between the lens plane andthe sensor plane such as to adjust the sharpness of the various areascovered in the ROI according to their importance for detecting theuser's physiological signals, discuss a fixed tilt between the lensplane and sensor plane, and discuss an adjustable electromechanicaltilting mechanism.

Various embodiments described herein involve an HMS that may beconnected, using wires and/or wirelessly, with a device carried by theuser and/or a non-wearable device. The HMS may include a battery, acomputer, sensors, and a transceiver.

FIG. 23a and FIG. 23b are schematic illustrations of possibleembodiments for computers (400, 410) that are able to realize one ormore of the embodiments discussed herein that include a “computer”. Thecomputer (400, 410) may be implemented in various ways, such as, but notlimited to, a microcontroller, a computer on a chip, a system-on-chip(SoC), a system-on-module (SoM), a processor with its requiredperipherals, a server computer, a client computer, a personal computer,a cloud computer, a network device, a handheld device (e.g., asmartphone), an head-mounted system (such as smartglasses, an augmentedreality system, a virtual reality system, and/or a smart-helmet), acomputing device embedded in a wearable device (e.g., a smartwatch or acomputer embedded in clothing), a computing device implanted in thehuman body, and/or any other computer form capable of executing a set ofcomputer instructions. Further, references to a computer or a processorinclude any collection of one or more computers and/or processors (whichmay be at different locations) that individually or jointly execute oneor more sets of computer instructions. This means that the singular term“computer” is intended to imply one or more computers, which jointlyperform the functions attributed to “the computer”. In particular, somefunctions attributed to the computer may be performed by a computer on awearable device (e.g., smartglasses) and/or a computer of the user(e.g., smartphone), while other functions may be performed on a remotecomputer, such as a cloud-based server.

The computer 400 includes one or more of the following components:processor 401, memory 402, computer readable medium 403, user interface404, communication interface 405, and bus 406. The computer 410 includesone or more of the following components: processor 411, memory 412, andcommunication interface 413.

Functionality of various embodiments may be implemented in hardware,software, firmware, or any combination thereof. If implemented at leastin part in software, implementing the functionality may involve acomputer program that includes one or more instructions or code storedor transmitted on a computer-readable medium and executed by one or moreprocessors. Computer-readable media may include computer-readablestorage media, which corresponds to a tangible medium such as datastorage media, and/or communication media including any medium thatfacilitates transfer of a computer program from one place to another.Computer-readable medium may be any media that can be accessed by one ormore computers to retrieve instructions, code, data, and/or datastructures for implementation of the described embodiments. A computerprogram product may include a computer-readable medium. In one example,the computer-readable medium 403 may include one or more of thefollowing: RAM, ROM, EEPROM, optical storage, magnetic storage, biologicstorage, flash memory, or any other medium that can store computerreadable data.

A computer program (also known as a program, software, softwareapplication, script, program code, or code) can be written in any formof programming language, including compiled or interpreted languages,declarative or procedural languages. The program can be deployed in anyform, including as a standalone program or as a module, component,subroutine, object, or another unit suitable for use in a computingenvironment. A computer program may correspond to a file in a filesystem, may be stored in a portion of a file that holds other programsor data, and/or may be stored in one or more files that may be dedicatedto the program. A computer program may be deployed to be executed on oneor more computers that are located at one or more sites that may beinterconnected by a communication network.

Computer-readable medium may include a single medium and/or multiplemedia (e.g., a centralized or distributed database, and/or associatedcaches and servers) that store one or more sets of instructions. Invarious embodiments, a computer program, and/or portions of a computerprogram, may be stored on a non-transitory computer-readable medium, andmay be updated and/or downloaded via a communication network, such asthe Internet. Optionally, the computer program may be downloaded from acentral repository, such as Apple App Store and/or Google Play.Optionally, the computer program may be downloaded from a repository,such as an open source and/or community run repository (e.g., GitHub).

At least some of the methods described herein are “computer-implementedmethods” that are implemented on a computer, such as the computer (400,410), by executing instructions on the processor (401, 411).Additionally, at least some of these instructions may be stored on anon-transitory computer-readable medium.

As used herein, references to “one embodiment” (and its variations) meanthat the feature being referred to may be included in at least oneembodiment of the invention. Moreover, separate references to “oneembodiment”, “some embodiments”, “another embodiment”, “still anotherembodiment”, etc., may refer to the same embodiment, may illustratedifferent aspects of an embodiment, and/or may refer to differentembodiments.

Some embodiments may be described using the verb “indicating”, theadjective “indicative”, and/or using variations thereof. Herein,sentences in the form of “X is indicative of Y” mean that X includesinformation correlated with Y, up to the case where X equals Y. Statingthat “X indicates Y” or “X indicating Y” may be interpreted as “X beingindicative of Y”. Additionally, sentences in the form of“provide/receive an indication indicating whether X happened” may referto any indication method, including but not limited to:sending/receiving a signal when X happened and not sending/receiving asignal when X did not happen, not sending/receiving a signal when Xhappened and sending/receiving a signal when X did not happen, and/orsending/receiving a first signal when X happened and sending/receiving asecond signal X did not happen.

Herein, “most” of something is defined as above 51% of the something(including 100% of the something). Both a “portion” of something and a“region” of something refer to a value between a fraction of thesomething and 100% of the something. For example, sentences in the formof a “portion of an area” may cover between 0.1% and 100% of the area.As another example, sentences in the form of a “region on the user'sforehead” may cover between the smallest area captured by a single pixel(such as 0.1% or 5% of the forehead) and 100% of the forehead. The word“region” refers to an open-ended claim language, and a camera said tocapture a specific region on the face may capture just a small part ofthe specific region, the entire specific region, and/or a portion of thespecific region together with additional region(s).

The terms “comprises”, “comprising”, “includes”, “including”, “has”,“having”, or any other variation thereof, indicate an open-ended claimlanguage that can include additional limitations. The “a” or “an” isemployed to describe one or more, and the singular also includes theplural unless it is obvious that it is meant otherwise. For example, “acomputer” refers to one or more computers, such as a combination of awearable computer that operates together with a cloud computer.

The phrase “based on” indicates an open-ended claim language, and is tobe interpreted as “based, at least in part, on”. Additionally, statingthat a value is calculated “based on X” and following that, in a certainembodiment, that the value is calculated “also based on Y”, means thatin the certain embodiment, the value is calculated based on X and Y.Variations of the terms “utilize” and “use” indicate an open-ended claimlanguage, such that sentences in the form of “detecting X utilizing Y”are intended to mean “detecting X utilizing at least Y”, and sentencesin the form of “use X to calculate Y” are intended to mean “calculate Ybased on X”.

The terms “first”, “second” and so forth are to be interpreted merely asordinal designations, and shall not be limited in themselves. Apredetermined value is a fixed value and/or a value determined any timebefore performing a calculation that compares a certain value with thepredetermined value. A value is also considered to be a predeterminedvalue when the logic, used to determine whether a threshold thatutilizes the value is reached, is known before start performingcomputations to determine whether the threshold is reached.

The embodiments of the invention may include any variety of combinationsand/or integrations of the features of the embodiments described herein.Although some embodiments may depict serial operations, the embodimentsmay perform certain operations in parallel and/or in different ordersfrom those depicted. Moreover, the use of repeated reference numeralsand/or letters in the text and/or drawings is for the purpose ofsimplicity and clarity and does not in itself dictate a relationshipbetween the various embodiments and/or configurations discussed. Theembodiments are not limited in their applications to the order of stepsof the methods, or to details of implementation of the devices, set inthe description, drawings, or examples. Moreover, individual blocksillustrated in the figures may be functional in nature and therefore maynot necessarily correspond to discrete hardware elements.

Certain features of the embodiments, which may have been, for clarity,described in the context of separate embodiments, may also be providedin various combinations in a single embodiment. Conversely, variousfeatures of the embodiments, which may have been, for brevity, describedin the context of a single embodiment, may also be provided separatelyor in any suitable sub-combination. Embodiments described in conjunctionwith specific examples are presented by way of example, and notlimitation. Moreover, it is evident that many alternatives,modifications, and variations will be apparent to those skilled in theart. It is to be understood that other embodiments may be utilized andstructural changes may be made without departing from the scope of theembodiments. Accordingly, this disclosure is intended to embrace allsuch alternatives, modifications, and variations that fall within thespirit and scope of the appended claims and their equivalents.

We claim:
 1. A system configured to calculate extent of congestive heartfailure (CHF), comprising: smartglasses configured to be worn on auser's head; an inward-facing camera, physically coupled to thesmartglasses, configured to capture images of an area comprising skin onthe user's head; wherein the area is larger than 4 cm{circumflex over( )}2, and the inward-facing camera is mounted more than 5 mm away fromthe user's head; a sensor, physically coupled to the smartglasses,configured to measure a signal indicative of a respiration rate of theuser; and a computer configured to calculate the extent of CHF based on:a facial blood flow pattern recognizable in the images, and therespiration rate of the user recognizable in the signal.
 2. The systemof claim 1, wherein the images and the signal are measured over a periodspanning multiple days, and the computer is further configured toidentify an exacerbation of the CHF based on: a reduction in averagefacial blood flow recognizable in the images taken during the period,and an increase in an average respiration rate recognizable in thesignal measured during the period.
 3. The system of claim 1, wherein thearea comprises a portion of the lips of the user.
 4. The system of claim1, wherein the images and the signal were measured while the user was atrest and prior to a period during which the user walked; wherein thecomputer is further configured to receive: (i) additional images, takenwithin ten minutes after the period with the inward-facing camera, and(ii) an additional signal indicative of an additional respiration rateof the user, measured with the sensor within ten minutes after theperiod; and wherein the computer is further configured to calculate theextent of CHF based on: a difference between the facial blood flowpattern recognizable in the images and an additional facial blood flowpattern recognizable in the additional images, and a difference betweenthe respiration rate recognizable in the signal and the additionalrespiration rate recognizable in the additional signal.
 5. The system ofclaim 4, further comprising a movement sensor, physically coupled to thesmartglasses, configured to measure movements of the user; wherein thecomputer is further configured to: calculate a number of steps performedby the user during the period, and to calculate the extent of CHFresponsive to the number of steps exceeding a predetermined thresholdgreater than twenty steps.
 6. The system of claim 4, wherein thecomputer is further configured to: calculate a value indicative of skincolor at different times based on the additional images, and tocalculate the extent of CHF based on a length of a duration followingthe period, in which the difference between the skin color and abaseline skin color, calculated based on the images, was above athreshold.
 7. The system of claim 4, wherein the computer is furtherconfigured to: calculate a value indicative of skin color at differenttimes based on the additional images, and to calculate the extent of CHFbased on a rate of return of the user's skin color to a baseline skincolor calculated based on the images.
 8. The system of claim 4, whereinthe computer is further configured to: calculate respiration rates ofthe user at different times based on the additional signal, and tocalculate the extent of CHF based on a length of a duration followingthe period, in which the difference between the respiration rate of theuser and a baseline respiration rate, calculated based on therespiration rates, was above a threshold.
 9. The system of claim 1,wherein the computer is further configured to calculate, based on theimages, a value indicative of an extent to which skin in the area isblue and/or gray, and to utilize a difference between the value and abaseline value for the extent to which skin in the area is blue and/orgray to calculate the extent of CHF; wherein the baseline value wasdetermined while the user experienced a certain baseline extent of CHF.10. The system of claim 1, wherein the computer is further configured tocalculate, based on the images, a value indicative of extent of colorchanges to skin in the area due to cardiac pulses, and to utilize adifference between the value and a baseline value for the extent of thecolor changes to calculate the extent of CHF; wherein the baseline valuefor the extent of the color changes was determined while the userexperienced a certain baseline extent of CHF.
 11. The system of claim 1,further comprising a head-mounted sensor configured to measuretemperature of a region comprising skin on the user's head (T_(skin));and the computer is further configured to utilize T_(skin) to compensatefor effects of skin temperature on the facial blood flow pattern. 12.The system of claim 1, further comprising a head-mounted sensorconfigured to measure environmental temperature (T_(env)); and thecomputer is further configured to utilize T_(env) to compensate foreffects of physiologic changes related to regulating the user's bodytemperature on the facial blood flow pattern.
 13. A system configured toidentify exacerbation of congestive heart failure (CHF), comprising:smartglasses configured to be worn on a user's head; an inward-facingcamera, physically coupled to the smartglasses, configured to captureimages of an area comprising skin on the user's head, which areindicative of a facial blood flow pattern of the user; wherein the areais larger than 4 cm{umlaut over ( )}2, and the inward-facing camera ismounted more than 5 mm away from the user's head; a sensor, physicallycoupled to the smartglasses, configured to measure a signal indicativeof a respiration rate of the user; and a computer configured to: receiveprevious images of the area, which are indicative of a previous facialblood flow pattern while the user had a certain extent of CHF; receive aprevious respiration rate taken while the user had the certain extent ofCHF; and identify exacerbation of the CHF based on: a difference above afirst predetermined threshold between the facial blood flow pattern andthe previous facial blood flow pattern, and an increase above a secondpredetermined threshold in the respiration rate compared to the previousrespiration rate.
 14. The system of claim 13, further comprising ahead-mounted sensor configured to measure temperature of a regioncomprising skin on the user's head (T_(skin)); wherein the computer isfurther configured to utilize T_(skin) to compensate for effects of skintemperature on the facial blood flow pattern.
 15. The system of claim13, further comprising a head-mounted sensor configured to measureenvironmental temperature (T_(env)); wherein the computer is furtherconfigured to utilize T_(env) to compensate for effects of physiologicchanges related to regulating the user's body temperature on the facialblood flow pattern.
 16. A system configured to calculate extent ofcongestive heart failure (CHF), comprising: smartglasses configured tobe worn on a user's head; an inward-facing camera, physically coupled tothe smartglasses, configured to capture images of an area comprisingskin on the user's head; wherein the area is larger than 4 cm{umlautover ( )}2, and the inward-facing camera is mounted more than 5 mm awayfrom the user's head; and a computer configured to: receive a first setof the images taken while the user was at rest and prior to a periodduring which the user performed physical activity; receive a second setof the images taken within ten minutes after the period; and calculatethe extent of CHF based on differences in facial blood flow patternsrecognizable in the first and second sets of the images.
 17. The systemof claim 16, further comprising a movement sensor, physically coupled tothe smartglasses, configured to measure movements of the user; whereinthe computer is further configured to detect the period during which theuser performed the physical activity based on the movements; and whereinthe physical activity comprises walking at least 20 steps.
 18. Thesystem of claim 16, wherein the computer is further configured tocalculate first and second series of heart rate values from portions ofiPPG signals extracted from the first and second sets of images,respectively; and wherein the computer is further configured tocalculate the extent of the CHF also based on the extent to which heartrate values in the second series were above heart rate values in thefirst series.
 19. The system of claim 18, wherein the computer isfurther configured to calculate the extent of CHF based on a durationafter the period in which the heart rate values in the second serieswere above the heart rate values in the first series.
 20. The system ofclaim 16, further comprising a head-mounted sensor configured to measuretemperature of a region comprising skin on the user's head (T_(skin));and the computer is further configured to utilize T_(skin) to compensatefor effects of skin temperature on the facial blood flow pattern.